2025-10-27 10:57:51.408 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:57:51.408 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:57:51.408 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:57:51.408 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:57:51.411 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:57:51.412 INFO 2458 --- [ocp-schedules-6,348e1d3405c1fedd,5ca234c0aec48240] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:57:51.425 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:57:51.426 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:57:51.426 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:57:51.427 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:57:51.428 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:57:51.428 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:57:51.429 INFO 2458 --- [ocp-schedules-3,01aa0aa71dbf2e33,e3e5f9f680b76443] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:57:51.430 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:57:51.431 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:57:51.431 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:57:51.431 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.431 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.432 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.432 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:57:51.432 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:57:51.439 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:57:51.442 INFO 2458 --- [ocp-schedules-27,5ab1f4cda2e2aa50,c21e0e575b2524cc] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 10:57:51.443 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.443 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:57:51.429 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:57:51.429 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:57:51.447 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:57:51.449 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:57:51.450 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:57:51.451 INFO 2458 --- [host-check-clockdiff1,01aa0aa71dbf2e33,50e2b8a3ec501f5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 10:57:51.452 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:57:51.452 INFO 2458 --- [host-check-clockdiff2,01aa0aa71dbf2e33,d8ec8c8e93394313] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 10:57:51.452 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:57:51.453 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:57:51.456 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:57:51.456 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:57:51.458 INFO 2458 --- [host-check-clockdiff3,01aa0aa71dbf2e33,64a78470b5d11a6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 10:57:51.465 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:57:51.466 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.466 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.466 INFO 2458 --- [host-check-clockdiff4,01aa0aa71dbf2e33,572c37410c92d381] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 10:57:51.467 INFO 2458 --- [host-check-clockdiff5,01aa0aa71dbf2e33,9e557df0f94f3d82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 10:57:51.474 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:57:51.474 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.475 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:57:51.475 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:57:51.478 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:57:51.510 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.511 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:57:51.517 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 10:57:51.517 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 10:57:51.519 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T10:57:51.518880263+08:00 2025-10-27 10:57:51.520 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T10:57:51.519513881+08:00 2025-10-27 10:57:51.520 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T10:57:51.519+08:00 2025-10-27 10:57:51.520 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T10:57:51.520+08:00 2025-10-27 10:57:51.521 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 10:57:51.523 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T10:57:51.521675681+08:00 2025-10-27 10:57:51.523 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 10:57:51.523 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T10:57:51.523+08:00 2025-10-27 10:57:51.526 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.526 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.527 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.538 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 10:57:51.538 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 10:57:51.538 INFO 2458 --- [host-check-agent-status1,348e1d3405c1fedd,5dc87bc8ab5af6c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 10:57:51.538 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T10:57:51.519+08:00 and status AVAILABLE 2025-10-27 10:57:51.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:57:51.539 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T10:57:51.523+08:00 and status AVAILABLE 2025-10-27 10:57:51.539 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T10:57:51.520+08:00 and status AVAILABLE 2025-10-27 10:57:51.540 INFO 2458 --- [host-check-clockdiff4,01aa0aa71dbf2e33,572c37410c92d381] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533871 0 0 , error= 2025-10-27 10:57:51.540 INFO 2458 --- [host-check-clockdiff2,01aa0aa71dbf2e33,d8ec8c8e93394313] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533871 -1 -1 , error= 2025-10-27 10:57:51.542 INFO 2458 --- [host-check-clockdiff1,01aa0aa71dbf2e33,50e2b8a3ec501f5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533871 0 0 , error= 2025-10-27 10:57:51.544 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 10:57:51.544 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.544 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.546 INFO 2458 --- [ocp-schedules-14,b2aa39ccaed89e63,3edd2b209f05f35e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 10:57:51.546 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.546 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 10:57:51.546 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 10:57:51.546 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:57:51.548 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 10:57:51.548 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 10:57:51.548 INFO 2458 --- [host-check-agent-status2,348e1d3405c1fedd,4b400d441c4f5e19] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 10:57:51.549 INFO 2458 --- [ocp-schedules-5,b728e3da3749d41d,dc65ca0c2b8735f4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 10:57:51.549 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 10:57:51.551 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 10:57:51.554 INFO 2458 --- [ocp-schedules-13,60a247d4400dec21,341c2cd17b2a0893] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 10:57:51.554 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:51:50.473, end=2025-10-27T10:56:51.292} 2025-10-27 10:57:51.554 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 1 dubious sqls 2025-10-27 10:57:51.556 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 10:57:51.556 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 10:57:51.556 INFO 2458 --- [host-check-agent-status3,348e1d3405c1fedd,fd5b7410e613b802] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 10:57:51.556 INFO 2458 --- [ocp-schedules-5,b728e3da3749d41d,dc65ca0c2b8735f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:57:51.557 INFO 2458 --- [ocp-schedules-14,b2aa39ccaed89e63,3edd2b209f05f35e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:57:51.558 INFO 2458 --- [ocp-schedules-5,b728e3da3749d41d,dc65ca0c2b8735f4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 10:57:51.559 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 10:57:51.561 INFO 2458 --- [ocp-schedules-13,60a247d4400dec21,341c2cd17b2a0893] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:57:51.562 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.563 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 10:57:51.563 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 10:57:51.563 INFO 2458 --- [host-check-agent-status4,348e1d3405c1fedd,f10e2c4d80987e7e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 10:57:51.564 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.565 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 10:57:51.565 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 10:57:51.566 INFO 2458 --- [ocp-schedules-18,1775b2f1cb8447a2,f675c9b974fb0200] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 10:57:51.566 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.567 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T10:57:51.567090306+08:00 2025-10-27 10:57:51.567 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.567 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T10:57:51.567+08:00 2025-10-27 10:57:51.568 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.568 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 10:57:51.568 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 10:57:51.568 INFO 2458 --- [host-check-agent-status5,348e1d3405c1fedd,1e0b7c9ed4250287] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 10:57:51.569 INFO 2458 --- [ocp-schedules-30,486a1923bebd6f39,16be2050e5aa2ca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 10:57:51.569 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.569 INFO 2458 --- [ocp-schedules-21,4956b90a7944da68,e4000925fc0308f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 10:57:51.570 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.573 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.573 INFO 2458 --- [ocp-schedules-22,2fd058abc6858d07,7dfefa70d5ed6012] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 10:57:51.574 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.574 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:57:51.575 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.576 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.577 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:57:51.578 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.578 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 10:57:51.579 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:57:51.579 INFO 2458 --- [ocp-schedules-22,2fd058abc6858d07,7dfefa70d5ed6012] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:57:51.579 INFO 2458 --- [ocp-schedules-21,4956b90a7944da68,e4000925fc0308f4] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 10:57:51.581 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:57:51.581 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:57:51.583 INFO 2458 --- [ocp-schedules-30,486a1923bebd6f39,16be2050e5aa2ca5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 10:57:51.585 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T10:57:51.567+08:00 and status AVAILABLE 2025-10-27 10:57:51.586 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:57:51.588 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 10:57:51.589 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T10:57:51.588187526+08:00 2025-10-27 10:57:51.589 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:57:51.590 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T10:57:51.589+08:00 2025-10-27 10:57:51.591 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.591 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.592 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 10:57:51.592 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.592 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:57:51.602 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T10:57:51.589+08:00 and status AVAILABLE 2025-10-27 10:57:51.608 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 10:57:51.622 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.623 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:57:51.624 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 10:57:51.632 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.632 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.632 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.633 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:57:51.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:57:51.636 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.636 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.636 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.636 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:57:51.640 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:57:51.644 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:57:51.649 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.650 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 10:57:51.659 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,34132cc06b2c3605,b05c9983030723b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=34132cc06b2c3605, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 10:57:51.659 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,34132cc06b2c3605,b05c9983030723b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=34132cc06b2c3605, duration=1 ms] 2025-10-27 10:57:51.661 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.662 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 10:57:51.667 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:57:51.669 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.675 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 10:57:51.678 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.678 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.678 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.679 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:57:51.690 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:57:51.697 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:57:51.699 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:51.700 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.700 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.702 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 10:57:51.707 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.707 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.707 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.707 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.707 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.708 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:57:51.710 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 10:57:51.711 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:51.711 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 10:57:51.711 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:51.711 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 10:57:51.712 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 10:57:51.714 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.714 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:57:51.715 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 10:57:51.715 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:57:51.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:57:51.720635+08:00, lastProcessTime=2025-10-27T10:57:41.717477+08:00 2025-10-27 10:57:51.722 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.722 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:57:51.726 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.726 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.726 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.726 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:57:51.730 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 10:57:51.731 INFO 2458 --- [ocp-schedules-14,b2aa39ccaed89e63,3edd2b209f05f35e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:57:51.733 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:57:51.760 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,3ea9dd4f4b6b1b64,4fa6a53a8c230abd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=3ea9dd4f4b6b1b64, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 10:57:51.760 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,3ea9dd4f4b6b1b64,4fa6a53a8c230abd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=3ea9dd4f4b6b1b64, duration=0 ms] 2025-10-27 10:57:51.763 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.763 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.763 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.764 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,b0c8f0f3771655f6,f5d98c6648277024] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b0c8f0f3771655f6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 10:57:51.764 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,b0c8f0f3771655f6,f5d98c6648277024] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b0c8f0f3771655f6, duration=1 ms] 2025-10-27 10:57:51.766 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 10:57:51.768 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 10:57:51.781 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 10:57:51.789 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.789 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.789 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.790 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:57:51.791 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 10:57:51.797 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.797 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.797 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.801 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 10:57:51.803 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.803 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.803 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.805 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:57:51.806 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 10:57:51.807 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.807 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.808 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.811 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 10:57:51.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.812 INFO 2458 --- [ocp-schedules-22,2fd058abc6858d07,7dfefa70d5ed6012] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 10:57:51.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.812 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.813 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:57:51.817 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:57:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:51.826 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.826 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:57:51.827 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 10:57:51.829 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.829 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.829 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.830 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:57:51.831 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 10:57:51.833 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:57:51.836 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 10:57:51.843 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:57:51.854 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 10:57:51.858 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 10:57:51.878 INFO 2458 --- [ocp-schedules-23,ca7282a7f88ab2a1,aab669cf9c1cbf7c] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 10:57:51.878 INFO 2458 --- [ocp-schedules-23,ca7282a7f88ab2a1,aab669cf9c1cbf7c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 10:57:51.881 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:57:51.904 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 10:57:51.904 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 10:57:51.912 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 10:57:51.914 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.914 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.915 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.917 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 10:57:51.917 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 10:57:51.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.918 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.920 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.921 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.923 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.923 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.923 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.924 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.925 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.925 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.925 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.925 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.926 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.927 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.927 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.927 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.928 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.929 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.929 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.929 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.932 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.933 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.933 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.933 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.936 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.937 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.937 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 10:57:51.939 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.939 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.939 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.940 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:57:51.941 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:57:51.941 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.941 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.941 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.943 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:57:51.943 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 10:57:51.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:57:51.947 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:51.947 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:57:51.948 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.948 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:57:51.950 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.952 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:57:51.953 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:57:51.954 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:57:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:57:51.964 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 10:57:51.966 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 10:57:51.977 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 10:57:51.979 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:57:51.986 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:51.986 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:57:51.994 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 10:57:52.033 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 10:57:52.035 INFO 2458 --- [host-check-host-status2,d39064c292a182c6,cf8e46ff158af257] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 10:57:52.038 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 10:57:52.043 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 10:57:52.045 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.049 INFO 2458 --- [host-check-host-status4,d39064c292a182c6,1ba7fdc54287b561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 10:57:52.074 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:57:52.085 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:57:52.086 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.086 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.086 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.087 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:57:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:57:52.088 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 10:57:52.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.090 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.093 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:57:51.720635+08:00 2025-10-27 10:57:52.094 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:57:52.094 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:57:52.095 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:57:52.096 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 10:57:52.099 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.099 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.099 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.099 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.099 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.100 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.100 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 10:57:52.100 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:57:52.102 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 10:57:52.111 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:57:52.114 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:57:52.125 INFO 2458 --- [pool-overview-container-4,7866d066a72aa3fe,bb88ed540786baba] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.130 INFO 2458 --- [host-check-host-status5,d39064c292a182c6,a31a1b0c5141ef94] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:57:52.132 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 10:57:52.133 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 10:57:52.136 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 10:57:52.136 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:57:52.137 INFO 2458 --- [ocp-schedules-24,cc814310af29f567,dd7d33d39927cae7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 10:57:52.142 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.142 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:57:52.146 INFO 2458 --- [host-check-host-status3,d39064c292a182c6,6550ba3d3bf65d0c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 10:57:52.161 INFO 2458 --- [host-check-host-status1,d39064c292a182c6,dcf69ede9310c622] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 10:57:52.179 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:57:52.188 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:57:52.223 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 10:57:52.223 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.224 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:57:52.226 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.227 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:57:52.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:57:52.245 INFO 2458 --- [pool-overview-container-2,116e3f1a189adc9e,8d4e68f7ce64aa14] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.266 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:57:52.290 INFO 2458 --- [pool-overview-container-5,54547be293abb366,f3773a1efec9dba3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.293 INFO 2458 --- [pool-overview-container-5,54547be293abb366,f3773a1efec9dba3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 10:57:52.299 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.299 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.300 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:57:52.307 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.307 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:57:52.331 INFO 2458 --- [pool-12-thread-12,116e3f1a189adc9e,9dd4f37372e8fbfb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 10:57:52.333 INFO 2458 --- [pool-12-thread-9,116e3f1a189adc9e,ed058ee5867f2d0d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 10:57:52.342 INFO 2458 --- [pool-12-thread-9,116e3f1a189adc9e,ed058ee5867f2d0d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 10:57:52.342 INFO 2458 --- [pool-12-thread-9,116e3f1a189adc9e,ed058ee5867f2d0d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 10:57:52.344 INFO 2458 --- [pool-12-thread-10,116e3f1a189adc9e,7d3f94360c6002dd] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 10:57:52.344 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:57:52.344 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 10:57:52.345 INFO 2458 --- [ForkJoinPool-7-worker-7,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.346 INFO 2458 --- [ForkJoinPool-7-worker-14,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.348 INFO 2458 --- [pool-12-thread-11,116e3f1a189adc9e,f1757dab3dd0efe1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 10:57:52.348 INFO 2458 --- [pool-12-thread-11,116e3f1a189adc9e,f1757dab3dd0efe1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 10:57:52.353 INFO 2458 --- [ForkJoinPool-7-worker-0,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.354 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 10:57:52.354 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 10:57:52.355 INFO 2458 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 10:57:52.356 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 10:57:52.356 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 10:57:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:57:52.383 INFO 2458 --- [pool-12-thread-11,116e3f1a189adc9e,f1757dab3dd0efe1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 10:57:52.388 INFO 2458 --- [pool-12-thread-11,116e3f1a189adc9e,f1757dab3dd0efe1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 10:57:52.396 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.396 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 10:57:52.397 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.398 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 10:57:52.399 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.401 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 10:57:52.409 INFO 2458 --- [host-check-clockdiff3,01aa0aa71dbf2e33,64a78470b5d11a6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533872 0 0 , error= 2025-10-27 10:57:52.415 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.415 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:57:52.416 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.416 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.416 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.418 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 10:57:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.424 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 10:57:52.430 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 10:57:52.431 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.432 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 10:57:52.433 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.433 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.433 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.433 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.434 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 10:57:52.435 INFO 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 10:57:52.438 INFO 2458 --- [ocp-async-10,63afdcce243e4503,1adc70381e1c6e88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:57:52.438 WARN 2458 --- [ocp-async-7,05793589260b5630,75b49a1bdec702a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:57:52.444 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.445 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 10:57:52.446 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.446 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 10:57:52.447 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 10:57:52.466 INFO 2458 --- [host-check-clockdiff5,01aa0aa71dbf2e33,9e557df0f94f3d82] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533872 0 0 , error= 2025-10-27 10:57:52.471 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.478 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 10:57:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.481 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:57:52.481 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 10:57:52.481 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.481 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.482 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.482 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.482 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.484 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 10:57:52.485 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 10:57:52.486 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.486 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.486 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 10:57:52.487 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.488 INFO 2458 --- [ForkJoinPool-7-worker-7,116e3f1a189adc9e,4a9013edfd960e27] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 10:57:52.492 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.494 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 10:57:52, 2025-10-27 10:57:52] 2025-10-27 10:57:52.496 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 10:57:52.503 WARN 2458 --- [ForkJoinPool-7-worker-9,116e3f1a189adc9e,c9f0769389714083] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 10:57:52.513 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:57:52.516 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.517 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:57:52.518 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 10:57:52.524 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 10:57:52.525 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.527 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 10:57:52, 2025-10-27 10:57:52] 2025-10-27 10:57:52.529 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 10:57:52.529 INFO 2458 --- [ForkJoinPool-7-worker-0,116e3f1a189adc9e,cba34917eda60573] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 10:57:52.529 INFO 2458 --- [pool-12-thread-10,116e3f1a189adc9e,7d3f94360c6002dd] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 10:57:52.546 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.546 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.547 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.547 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.547 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.552 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 10:57:52.560 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 10:57:52.561 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.562 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 10:57:52.563 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.565 INFO 2458 --- [ForkJoinPool-7-worker-14,116e3f1a189adc9e,9e43084fff1c50e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 10:57:52.568 INFO 2458 --- [pool-12-thread-12,116e3f1a189adc9e,9dd4f37372e8fbfb] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 10:57:52.576 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.581 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.585 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 10:57:52.589 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:57:52.644 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 10:57:52.644 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.644 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.644 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.645 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:57:52.647 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.647 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.647 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.648 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:57:52.690 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:57:52.691 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.691 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.691 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.691 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.691 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.692 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.696 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:57:52.734 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:57:52.735 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:57:52.789 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:57:52.789 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533402507000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 10:57:52.789 INFO 2458 --- [ocp-schedules-13,60a247d4400dec21,341c2cd17b2a0893] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:57:52.848 INFO 2458 --- [ocp-schedules-18,2a6f0c88f7f0cd84,d0c94030128fa44c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:57:52.851 INFO 2458 --- [ocp-schedules-18,2a6f0c88f7f0cd84,d0c94030128fa44c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:57:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 10:57:53.218 INFO 2458 --- [ocp-schedules-23,a810c0f351ff0b06,e46997a084f6c507] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20265,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5066,"seriesIdKeySize":8767,"secondWriteQueue":0,"minuteWriteCacheSize":7850} 2025-10-27 10:57:53.218 INFO 2458 --- [ocp-schedules-23,a810c0f351ff0b06,e46997a084f6c507] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 10:57:53.707 INFO 2458 --- [prometheus-minute-schedule-1,fe07915021aaf969,059d4e247a5546de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533880, interval=60 2025-10-27 10:57:53.716 INFO 2458 --- [prometheus-minute-schedule-1,fe07915021aaf969,059d4e247a5546de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:57:53.799 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:57:53.800 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:57:53.965 INFO 2458 --- [ocp-schedules-4,5c6afa30d99ef85d,978c3a6573da243b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 10:57:54.270 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 10:57:54.438 INFO 2458 --- [ocp-async-5,e74688e81db6e62d,c98359172d237cba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:57:54.438 WARN 2458 --- [ocp-async-1,67f0d14d528ceb84,bccd02ad4dcc724d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:57:55.396 INFO 2458 --- [ocp-schedules-10,7789459a1b52a5c2,22d1ad78add03ccd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:57:56.438 INFO 2458 --- [ocp-async-6,701cde2aa2981f6b,23b0e6b33aaa8c9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:57:56.438 WARN 2458 --- [ocp-async-9,81b4732e2fad9713,06da39ebbf284fb5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:57:56.707 INFO 2458 --- [prometheus-minute-schedule-1,b165f445e6850738,f891eccd05e53403] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533880, interval=60 2025-10-27 10:57:56.714 INFO 2458 --- [prometheus-minute-schedule-1,b165f445e6850738,f891eccd05e53403] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:57:56.800 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:57:56.801 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:57:57.426 INFO 2458 --- [ocp-schedules-13,6745cccbc590ca5f,88186106da7e3ba6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 10:57:57.433 INFO 2458 --- [ocp-schedules-13,6745cccbc590ca5f,88186106da7e3ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:57:57.475 INFO 2458 --- [ocp-schedules-13,6745cccbc590ca5f,88186106da7e3ba6] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 10:57:58.438 INFO 2458 --- [ocp-async-4,b03ee3c01f99553f,fae806ab2001da69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:57:58.438 WARN 2458 --- [ocp-async-3,97e7ce54b1f16de7,88cc562703b17c0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:57:58.500 INFO 2458 --- [ocp-async-10,91dbe4cd1836fe20,f33697e34a065c66] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:57:58.904 INFO 2458 --- [ocp-schedules-13,82b77e1c14e1f968,79da4fdc5964f22e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:57:58.936 WARN 2458 --- [scanner-112106515-1,82b77e1c14e1f968,a25780c5ee1fe025] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:57:58.979 INFO 2458 --- [alarm-detect-13,82b77e1c14e1f968,8a1b983b03bc1098] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761533808 ~ 1761533868 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533820, value=202154.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202154.56, values={1761533820=202154.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 10:57:59.011 INFO 2458 --- [ocp-schedules-15,a0dc16ef38cc24f9,af973d46cfc6f240] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 10:57:59.043 INFO 2458 --- [alarm-detect-2,82b77e1c14e1f968,9f3b980ec95e0cc1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761533809 ~ 1761533869 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533820, value=92.879)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.879, values={1761533820=92.879}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 10:57:59.065 INFO 2458 --- [alarm-detect-1,82b77e1c14e1f968,0a3e77f129fcad08] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761533809 ~ 1761533869 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533820, value=202155.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202155.912, values={1761533820=202155.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 10:57:59.076 INFO 2458 --- [alarm-detect-7,82b77e1c14e1f968,6287378576e3548a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533850 ~ 1761533869 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533865, value=42.872)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533865, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533865=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:57:59.564 INFO 2458 --- [ocp-schedules-4,b088c26cc47f7ebc,f896e63962afe4b2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 10:57:59.568 INFO 2458 --- [ocp-schedules-4,b088c26cc47f7ebc,f896e63962afe4b2] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 10:57:59.707 INFO 2458 --- [prometheus-minute-schedule-1,fe895abd17f34399,2660eb9625894db4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533880, interval=60 2025-10-27 10:57:59.714 INFO 2458 --- [prometheus-minute-schedule-1,fe895abd17f34399,2660eb9625894db4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:57:59.801 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:57:59.802 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:00.438 INFO 2458 --- [ocp-async-1,92e54623a7732757,fafebb2a9ca9706f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:00.439 WARN 2458 --- [ocp-async-5,5dcd50a038f6ba1e,e683a8fe0ac816f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:01.268 INFO 2458 --- [ocp-schedules-5,6815884b696fb843,df08ee870e295672] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 10:58:01.273 INFO 2458 --- [ocp-schedules-5,6815884b696fb843,df08ee870e295672] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:01.308 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:01.309 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 10:58:01.331 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:01.331 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:01.332 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:01.332 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:01.332 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:01.333 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 10:58:01.350 INFO 2458 --- [ocp-schedules-5,6815884b696fb843,df08ee870e295672] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 10:58:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:01.714291+08:00, lastProcessTime=2025-10-27T10:57:51.720635+08:00 2025-10-27 10:58:01.954 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:01.714291+08:00 2025-10-27 10:58:02.439 INFO 2458 --- [ocp-async-2,4fde53bae9c338ae,9a3e7e11c10db84e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:02.439 WARN 2458 --- [ocp-async-6,2d113d913fc38662,69c3a0c655b2d25d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:02.707 INFO 2458 --- [prometheus-minute-schedule-1,55c1a64d45493545,b79c7928deac230e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:02.715 INFO 2458 --- [prometheus-minute-schedule-1,55c1a64d45493545,b79c7928deac230e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:02.802 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:02.804 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:02.858 INFO 2458 --- [ocp-schedules-8,548a1dd28ca4804d,d0425c1f77e1f1ad] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:02.862 INFO 2458 --- [ocp-schedules-8,548a1dd28ca4804d,d0425c1f77e1f1ad] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:04.439 WARN 2458 --- [ocp-async-4,fe17ae8c08d916fa,1e01229059ab2af5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:04.439 INFO 2458 --- [ocp-async-7,c6d7413a067e9d9f,548725a113a4060e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:05.404 INFO 2458 --- [ocp-schedules-12,30501e9caf1b2492,073693987b1eb5d8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:05.707 INFO 2458 --- [prometheus-minute-schedule-1,9d35c86525de2e09,0516a31f902fd728] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:05.714 INFO 2458 --- [prometheus-minute-schedule-1,9d35c86525de2e09,0516a31f902fd728] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:05.804 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:05.804 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 10:58:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 10:58:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 10:58:06.058 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 10:58:06.099 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 10:58:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 10:58:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 10:58:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 10:58:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 10:58:06.126 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 10:58:06.127 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:06.127 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 10:58:06.129 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:06.130 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 10:58:06.237 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T10:55:30, start=2025-10-27T10:52:30, end=2025-10-27T10:55:00, updates=318, granule=120000ms, delay=180000ms 2025-10-27 10:58:06.245 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T10:55:30, target=2025-10-27T10:55:00, granule=120000ms, delay=30000ms) 2025-10-27 10:58:06.245 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T10:55:00, target=2025-10-27T10:42:30, granule=600000ms, delay=750000ms) 2025-10-27 10:58:06.245 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=241ms, period=120000ms 2025-10-27 10:58:06.369 INFO 2458 --- [ocp-schedules-24,e96c0d4d436ce2ff,e4c3e170a71eee63] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:58:06.377 INFO 2458 --- [ocp-schedules-24,e96c0d4d436ce2ff,e4c3e170a71eee63] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:06.377 INFO 2458 --- [ocp-schedules-24,e96c0d4d436ce2ff,e4c3e170a71eee63] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:06.439 INFO 2458 --- [ocp-async-5,40c583d88dc01dfd,453e4769a56cc53f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:06.439 WARN 2458 --- [ocp-async-8,9dafe50a3d9a0ddc,eeeaa50c31e8089c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:08.439 WARN 2458 --- [ocp-async-9,e7c2273125454196,7f6b4e4fb8095ec2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:08.439 INFO 2458 --- [ocp-async-6,14672de8204a598b,0dd21514129edcfd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:08.501 INFO 2458 --- [ocp-async-4,2279695edb4796e9,a3a58a132339c8fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:08.707 INFO 2458 --- [prometheus-minute-schedule-1,bfa1b8d09861a85d,c4562e41451de9f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:08.715 INFO 2458 --- [prometheus-minute-schedule-1,bfa1b8d09861a85d,c4562e41451de9f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:08.732 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,09cab954ba954cbf,4fa65ce473c7b548] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=09cab954ba954cbf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 10:58:08.732 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,09cab954ba954cbf,4fa65ce473c7b548] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=09cab954ba954cbf, duration=0 ms] 2025-10-27 10:58:08.805 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:08.805 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:09.081 INFO 2458 --- [ocp-schedules-11,3cb574a5172346ec,3a1410f650977ef9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:58:09.090 WARN 2458 --- [scanner-112106745-1,3cb574a5172346ec,2df714ae082b436b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:58:10.013 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 10:58:10.013 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 10:58:10.052 INFO 2458 --- [Check obproxy1,1263dd742b76890a,6941e924a88e45ba] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:58:10.052 INFO 2458 --- [Check obproxy1,1263dd742b76890a,6941e924a88e45ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:58:10.053 INFO 2458 --- [Collect obproxy1,f035e4df87fc1883,01c4422a491f5c3f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:58:10.054 INFO 2458 --- [Collect obproxy1,f035e4df87fc1883,01c4422a491f5c3f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:58:10.083 INFO 2458 --- [Check obproxy1,1263dd742b76890a,6941e924a88e45ba] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 10:58:10.086 INFO 2458 --- [Check obproxy1,1263dd742b76890a,6941e924a88e45ba] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 10:58:10.086 INFO 2458 --- [Check obproxy1,1263dd742b76890a,6941e924a88e45ba] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 10:58:10.088 INFO 2458 --- [Collect obproxy1,f035e4df87fc1883,01c4422a491f5c3f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 10:58:10.091 INFO 2458 --- [Collect obproxy1,f035e4df87fc1883,01c4422a491f5c3f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 10:58:10.094 INFO 2458 --- [Collect obproxy1,f035e4df87fc1883,01c4422a491f5c3f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 10:58:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 10:58:10.439 WARN 2458 --- [ocp-async-3,e4bf3857325d652d,53128b6c13bbc345] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:10.439 INFO 2458 --- [ocp-async-10,5f47ec996b81a46f,4f434498f24ac434] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:10.817 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811} 2025-10-27 10:58:10.825 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView= 2025-10-27 10:58:10.826 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:10.831 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:10.831 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 10:58:10.831 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:10.950 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447490950000, endTimeUs=1761533770811000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T02:58:10.950Z, endTime=2025-10-27T02:56:10.811Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 10:58:10.961 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 10:58:10.961 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 10:58:10.986 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 10:58:10.986 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533770811000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 10:58:10.995 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 10:58:10.995 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 10:58:11.020 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 10:58:11.020 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533770811000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 10:58:11.024 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 10:58:11.024 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T10:58:10.95, end=2025-10-27T10:56:10.811}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 10:58:11.025 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:11.027 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:11.027 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 10:58:11.027 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:11.029 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:11.031 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:11.031 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 10:58:11.031 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:11.033 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:11.035 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:11.035 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 10:58:11.035 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:11.037 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:11.039 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:11.039 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 10:58:11.039 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:11.044 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 10:58:11.046 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:54:09.968, end=2025-10-27T10:56:10.811},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 10:58:11.046 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 10:58:11.046 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 10:58:11.048 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 10:58:11.707 INFO 2458 --- [prometheus-minute-schedule-1,51d17d53f534f569,391ba80a8888b156] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:11.715 INFO 2458 --- [prometheus-minute-schedule-1,51d17d53f534f569,391ba80a8888b156] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:11.716427+08:00, lastProcessTime=2025-10-27T10:58:01.714291+08:00 2025-10-27 10:58:11.805 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:11.806 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:11.822 INFO 2458 --- [ocp-schedules-8,40f204031e2673f5,68bb245df7c2fea6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 10:58:11.822 INFO 2458 --- [ocp-schedules-8,40f204031e2673f5,68bb245df7c2fea6] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 10:58:11.829 INFO 2458 --- [ocp-schedules-8,40f204031e2673f5,68bb245df7c2fea6] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:58:11.829 INFO 2458 --- [ocp-schedules-8,40f204031e2673f5,68bb245df7c2fea6] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:58:11.925 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:11.716427+08:00 2025-10-27 10:58:12.439 WARN 2458 --- [ocp-async-10,0f3b9fee49449d6a,577fc766cb0e235a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:12.439 INFO 2458 --- [ocp-async-7,7d3687cedf3cda97,1f523a980b2d19f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:12.870 INFO 2458 --- [ocp-schedules-9,65f3d532a1a4384e,1771e21cacb3b6a3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:12.874 INFO 2458 --- [ocp-schedules-9,65f3d532a1a4384e,1771e21cacb3b6a3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:13.061 INFO 2458 --- [ocp-schedules-14,f5907eede7dba805,0a0d2c38c3161345] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 10:58:13.061 INFO 2458 --- [ocp-schedules-14,f5907eede7dba805,0a0d2c38c3161345] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 10:58:13.116 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 10:58:13.116 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 10:58:13.116 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 10:58:13.116 INFO 2458 --- [Collect all tenants major compactions6,f5907eede7dba805,712170ccfcabf471] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 10:58:13.116 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 10:58:13.117 INFO 2458 --- [Collect all tenants major compactions5,f5907eede7dba805,1f5d2b649c258059] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 10:58:13.153 INFO 2458 --- [Collect all tenants major compactions5,f5907eede7dba805,1f5d2b649c258059] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.154 INFO 2458 --- [Collect all tenants major compactions6,f5907eede7dba805,712170ccfcabf471] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.155 INFO 2458 --- [Collect all tenants major compactions6,f5907eede7dba805,712170ccfcabf471] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 10:58:13.155 INFO 2458 --- [Collect all tenants major compactions5,f5907eede7dba805,1f5d2b649c258059] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 10:58:13.158 INFO 2458 --- [Collect all tenants major compactions5,f5907eede7dba805,1f5d2b649c258059] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 10:58:13.158 INFO 2458 --- [Collect all tenants major compactions6,f5907eede7dba805,712170ccfcabf471] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 10:58:13.166 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.166 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.166 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.166 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.167 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.168 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 10:58:13.170 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.170 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:13.171 INFO 2458 --- [Collect all tenants major compactions3,f5907eede7dba805,11016e292b4dde7d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 10:58:13.172 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 10:58:13.172 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 10:58:13.173 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 10:58:13.175 INFO 2458 --- [Collect all tenants major compactions4,f5907eede7dba805,9e9d585b51859e82] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 10:58:13.177 INFO 2458 --- [Collect all tenants major compactions2,f5907eede7dba805,cb66770ae7ad6a69] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 10:58:13.177 INFO 2458 --- [Collect all tenants major compactions1,f5907eede7dba805,30dcf678623087d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 10:58:13.178 INFO 2458 --- [ocp-schedules-14,f5907eede7dba805,0a0d2c38c3161345] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 10:58:13.380 INFO 2458 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761533820, nodeCount=817, affectRows=817 2025-10-27 10:58:13.705 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835370602979598336-1761533760' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835370602979598336-1761533760' for key 'PRIMARY' 2025-10-27 10:58:13.713 INFO 2458 --- [ocp-schedules-6,ff0bacac54f8ba2d,df02833a5ca64e25] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T10:58:13.713+08:00, clusters=[1, 2] 2025-10-27 10:58:13.715 INFO 2458 --- [ocp-schedules-6,ff0bacac54f8ba2d,df02833a5ca64e25] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 10:58:13.717 INFO 2458 --- [ocp-schedules-6,ff0bacac54f8ba2d,df02833a5ca64e25] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 10:58:13.764 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=1000, affectRows=1000 2025-10-27 10:58:13.767 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=1000, affectRows=1000 2025-10-27 10:58:13.768 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=994, validCount=992 2025-10-27 10:58:13.769 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=1000, affectRows=1000 2025-10-27 10:58:13.771 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=996, validCount=992 2025-10-27 10:58:13.771 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=1000, affectRows=1000 2025-10-27 10:58:13.773 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=996, validCount=992 2025-10-27 10:58:13.775 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=999, validCount=995 2025-10-27 10:58:13.790 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=66, affectRows=66 2025-10-27 10:58:13.790 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=65, validCount=65 2025-10-27 10:58:14.440 INFO 2458 --- [ocp-async-2,bcd0ec1727d856b7,6f44b1c360cc98db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:14.440 WARN 2458 --- [ocp-async-5,9b35960ddcbb4192,3d4d0dab79aa1e68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:14.707 INFO 2458 --- [prometheus-minute-schedule-1,0583d29f26e9f30f,8c613fb1102fae2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:14.715 INFO 2458 --- [prometheus-minute-schedule-1,0583d29f26e9f30f,8c613fb1102fae2a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:14.806 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:14.807 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:15.411 INFO 2458 --- [ocp-schedules-11,291eb90b69a80c30,6efafaa9a8612bc2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:16.440 WARN 2458 --- [ocp-async-3,2148494331ff0bd9,a2d6a380c062f4fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:16.440 INFO 2458 --- [ocp-async-2,5df510da4f2ff97f,32f686c454e7d35a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:17.519 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533820, nodeCount=1000, affectRows=1000 2025-10-27 10:58:17.523 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533820, nodeCount=1000, validCount=988 2025-10-27 10:58:17.707 INFO 2458 --- [prometheus-minute-schedule-1,790a0b09962ac4f8,0c9d2d2eb056c0df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:17.715 INFO 2458 --- [prometheus-minute-schedule-1,790a0b09962ac4f8,0c9d2d2eb056c0df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:17.807 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:17.808 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:18.440 WARN 2458 --- [ocp-async-2,6744faaaf4069844,1e1401c4f08bd2fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:18.440 INFO 2458 --- [ocp-async-2,d4751d08b6b4a4a2,a4ac0e91143d7d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:18.501 INFO 2458 --- [ocp-async-5,6e7ed521c72fd8f6,6aa581166da7e04c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:18.638 INFO 2458 --- [ocp-schedules-29,82388976765704bf,b67e9bf209841d92] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 10:58:18.741 INFO 2458 --- [ocp-schedules-22,2f0465e8de2fd343,9fbc3ad353cd5213] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:18.741 INFO 2458 --- [ocp-schedules-22,2f0465e8de2fd343,9fbc3ad353cd5213] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 10:58:19.217 INFO 2458 --- [ocp-schedules-5,083dd520c02f56f2,d4ac842be8cb8d3a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:58:19.270 WARN 2458 --- [scanner-112108458-1,083dd520c02f56f2,c3be0e444275c2f1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:58:19.401 INFO 2458 --- [alarm-detect-3,083dd520c02f56f2,6a1956dfbd763b16] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533865 ~ 1761533889 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533880, value=44.315)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533880, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533880=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:58:20.440 WARN 2458 --- [ocp-async-5,ca2e9722fe22dcd8,b2d6feec80d9afa0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:20.440 INFO 2458 --- [ocp-async-6,a6e10d9aa203330d,b10f983b773ddf46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:20.707 INFO 2458 --- [prometheus-minute-schedule-1,d1d42bb549616da8,343e5d385878e616] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:20.715 INFO 2458 --- [prometheus-minute-schedule-1,d1d42bb549616da8,343e5d385878e616] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:20.808 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:20.809 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:21.408 INFO 2458 --- [ocp-schedules-7,f35f57cd4630e527,a2dd62beb3252965] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:21.408 INFO 2458 --- [ocp-schedules-3,9bd444bb0104c3d7,84325bd1df9da3d7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:58:21.409 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:58:21.410 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:58:21.411 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:58:21.411 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:58:21.426 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:21.427 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:21.427 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:21.427 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:21.427 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:21.428 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:21.428 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:21.429 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:21.431 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:21.432 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:21.432 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:21.432 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:21.432 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:21.433 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:21.433 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:21.434 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:21.434 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:21.437 INFO 2458 --- [ocp-schedules-29,fbbff487e8dfa9b9,d1190f33e76f1745] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:21.438 INFO 2458 --- [host-check-clockdiff1,fbbff487e8dfa9b9,51a2da0e9525587b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 10:58:21.438 INFO 2458 --- [host-check-clockdiff2,fbbff487e8dfa9b9,588f2adcc05692f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 10:58:21.438 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:21.440 INFO 2458 --- [host-check-clockdiff4,fbbff487e8dfa9b9,bc5d39e1d1acd57f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 10:58:21.441 INFO 2458 --- [host-check-clockdiff5,fbbff487e8dfa9b9,253aebfb2ec77f43] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 10:58:21.442 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:21.442 INFO 2458 --- [ocp-schedules-24,2706b046214b5b7e,ffbd7a390cbde64b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 10:58:21.443 INFO 2458 --- [host-check-clockdiff3,fbbff487e8dfa9b9,859fb13e1569366f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 10:58:21.445 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:21.455 INFO 2458 --- [host-check-clockdiff4,fbbff487e8dfa9b9,bc5d39e1d1acd57f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533901 0 0 , error= 2025-10-27 10:58:21.455 INFO 2458 --- [host-check-clockdiff3,fbbff487e8dfa9b9,859fb13e1569366f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533901 -1 -1 , error= 2025-10-27 10:58:21.468 INFO 2458 --- [host-check-clockdiff2,fbbff487e8dfa9b9,588f2adcc05692f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533901 -1 -1 , error= 2025-10-27 10:58:21.468 INFO 2458 --- [host-check-clockdiff1,fbbff487e8dfa9b9,51a2da0e9525587b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533901 0 0 , error= 2025-10-27 10:58:21.507 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 10:58:21.507 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 10:58:21.507 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 10:58:21.510 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T10:58:21.509611048+08:00 2025-10-27 10:58:21.512 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 10:58:21.512 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 10:58:21.512 INFO 2458 --- [host-check-agent-status1,9bd444bb0104c3d7,93897a99e2ff2d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 10:58:21.512 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T10:58:21.510+08:00 2025-10-27 10:58:21.511 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T10:58:21.510423820+08:00 2025-10-27 10:58:21.513 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T10:58:21.513+08:00 2025-10-27 10:58:21.514 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 10:58:21.518 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 10:58:21.518 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 10:58:21.518 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 10:58:21.518 INFO 2458 --- [host-check-agent-status5,9bd444bb0104c3d7,8a608b107aee651b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 10:58:21.519 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T10:58:21.510+08:00 and status AVAILABLE 2025-10-27 10:58:21.519 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 10:58:21.520 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T10:58:21.518934295+08:00 2025-10-27 10:58:21.521 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T10:58:21.520+08:00 2025-10-27 10:58:21.521 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 10:58:21.522 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 10:58:21.522 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 10:58:21.522 INFO 2458 --- [host-check-agent-status3,9bd444bb0104c3d7,df78271ace6ca5da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 10:58:21.522 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T10:58:21.513+08:00 and status AVAILABLE 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T10:58:21.524543587+08:00 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-agent-status2,9bd444bb0104c3d7,65711ddaf8ef67cb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 10:58:21.524 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 10:58:21.525 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T10:58:21.524+08:00 2025-10-27 10:58:21.526 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 10:58:21.526 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 10:58:21.527 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T10:58:21.527158767+08:00 2025-10-27 10:58:21.528 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T10:58:21.527+08:00 2025-10-27 10:58:21.530 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T10:58:21.520+08:00 and status AVAILABLE 2025-10-27 10:58:21.534 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T10:58:21.524+08:00 and status AVAILABLE 2025-10-27 10:58:21.534 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 10:58:21.534 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 10:58:21.537 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 10:58:21.537 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 10:58:21.537 INFO 2458 --- [host-check-agent-status4,9bd444bb0104c3d7,4c87330cefe97225] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 10:58:21.538 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T10:58:21.527+08:00 and status AVAILABLE 2025-10-27 10:58:21.538 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 10:58:21.542 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 10:58:21.543 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:21.545 INFO 2458 --- [ocp-schedules-23,7ae376b13074db80,da2407ff15ab0ec1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 10:58:21.546 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:21.549 INFO 2458 --- [ocp-schedules-26,4d192a829ee8a325,f9111296a6cbba99] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 10:58:21.554 INFO 2458 --- [ocp-schedules-30,2a4b2f234ca0b4bc,5c55008a16174fac] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 10:58:21.555 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:21.556 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:21.558 INFO 2458 --- [ocp-schedules-23,7ae376b13074db80,da2407ff15ab0ec1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:21.559 INFO 2458 --- [ocp-schedules-30,2a4b2f234ca0b4bc,5c55008a16174fac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:21.565 INFO 2458 --- [ocp-schedules-26,4d192a829ee8a325,f9111296a6cbba99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:21.566 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:21.567 INFO 2458 --- [ocp-schedules-26,4d192a829ee8a325,f9111296a6cbba99] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 10:58:21.569 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:21.575 INFO 2458 --- [ocp-schedules-9,eba0c79d129c0922,6b025390240d133b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 10:58:21.576 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:21.581 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:21.586 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:21.588 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.602 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 10:58:21.622 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:21.625 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:21.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.635 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:21.639 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 10:58:21.639 INFO 2458 --- [ocp-schedules-23,7ae376b13074db80,da2407ff15ab0ec1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:58:21.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:21.641 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 10:58:21.642 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 10:58:21.659 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:21.660 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 10:58:21.661 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:21.671 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 10:58:21.673 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 10:58:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:21.677 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 10:58:21.689 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 10:58:21.693 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 10:58:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:21.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:58:21.705 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:21.706 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:21.709 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 10:58:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:21.714698+08:00, lastProcessTime=2025-10-27T10:58:11.716427+08:00 2025-10-27 10:58:21.715 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 10:58:21.720 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 10:58:21.727 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:21.728 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 10:58:21.745 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:21.749 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 10:58:21.753 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:21.755 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 10:58:21.760 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 10:58:21.763 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 10:58:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.764 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.764 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:21.766 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.766 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:58:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.771 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:58:21.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:21.781 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 10:58:21.784 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 10:58:21.788 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:21.794 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:21.803 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 10:58:21.805 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 10:58:21.808 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:21.808 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:21.811 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:21.817 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 10:58:21.817 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 10:58:21.831 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 10:58:21.833 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:21.836 INFO 2458 --- [host-check-host-status4,f35f57cd4630e527,2ff858905e0f893d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 10:58:21.841 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:21.845 INFO 2458 --- [host-check-host-status3,f35f57cd4630e527,3eb248a644d5d961] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 10:58:21.860 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:21.864 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 10:58:21.872 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:21.873 INFO 2458 --- [host-check-host-status2,f35f57cd4630e527,3ad329ea6001f2c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 10:58:21.873 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:21.875 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 10:58:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:21.879 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:58:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:21.889 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 10:58:21.892 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 10:58:21.900 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:21.903 INFO 2458 --- [host-check-host-status5,f35f57cd4630e527,5f69a3cc311f70ed] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 10:58:21.903 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:21.924 INFO 2458 --- [host-check-host-status1,f35f57cd4630e527,53c3fa1d12ec77e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 10:58:21.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:21.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:21.964 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:58:21.968 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:21.714698+08:00 2025-10-27 10:58:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:22.006 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:22.006 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:22.006 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 10:58:22.006 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.007 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:58:22.009 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.010 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:58:22.026 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:58:22.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:22.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:22.054 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:22.055 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:58:22.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:22.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:58:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 10:58:22.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:22.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 10:58:22.168 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 10:58:22.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:22.180 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:22.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:58:22.290 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 10:58:22.291 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.291 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.291 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.292 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:58:22.293 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.293 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.294 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:58:22.333 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:58:22.379 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:22.380 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.380 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.380 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:22.380 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:22.380 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:22.381 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:58:22.390 INFO 2458 --- [host-check-clockdiff5,fbbff487e8dfa9b9,253aebfb2ec77f43] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533902 1 1 , error= 2025-10-27 10:58:22.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:58:22.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533402507000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 10:58:22.438 INFO 2458 --- [ocp-schedules-30,2a4b2f234ca0b4bc,5c55008a16174fac] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:58:22.440 WARN 2458 --- [ocp-async-4,297989f4c33bd434,7b675491371c586f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:22.440 INFO 2458 --- [ocp-async-8,62ffdb229803df15,1b95f9950305c38c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:22.881 INFO 2458 --- [ocp-schedules-22,e9d01c492e69f734,84453b671b6e8c00] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:22.885 INFO 2458 --- [ocp-schedules-22,e9d01c492e69f734,84453b671b6e8c00] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:23.707 INFO 2458 --- [prometheus-minute-schedule-1,a8ad243eca98c03e,1a96726896d903ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:23.717 INFO 2458 --- [prometheus-minute-schedule-1,a8ad243eca98c03e,1a96726896d903ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:23.809 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:23.809 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:24.440 WARN 2458 --- [ocp-async-2,ddb4250f7e864732,238f983d244e043e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:24.440 INFO 2458 --- [ocp-async-10,2a4e2fa0021ffa65,76e00f004ddbdbfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:25.418 INFO 2458 --- [ocp-schedules-20,c794b7dbf513859e,8008a5d8e9abf3fd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:25.573 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,778661887c9c550f,abd9c0d494fc62a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /pre_login_loading?null, client=10.40.198.13, traceId=778661887c9c550f, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.WebController.preLoginLoading(), args=] 2025-10-27 10:58:25.573 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,778661887c9c550f,abd9c0d494fc62a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /pre_login_loading client=10.40.198.13, traceId=778661887c9c550f, duration=17 ms] 2025-10-27 10:58:26.380 INFO 2458 --- [ocp-schedules-7,56d88f6700daf3f8,d9964c10021b6235] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:58:26.389 INFO 2458 --- [ocp-schedules-7,56d88f6700daf3f8,d9964c10021b6235] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:26.389 INFO 2458 --- [ocp-schedules-7,56d88f6700daf3f8,d9964c10021b6235] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:26.440 INFO 2458 --- [ocp-async-10,3c63b0b02745f516,09dfd85458e63f82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:26.440 WARN 2458 --- [ocp-async-7,90db6179ecf53ced,ead3e81f8713394c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:26.602 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,66d1be4f6825c859,5292ef7995c69d51] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.40.198.13, traceId=66d1be4f6825c859, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-10-27 10:58:26.625 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,66d1be4f6825c859,5292ef7995c69d51] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.40.198.13, traceId=66d1be4f6825c859, duration=28 ms] 2025-10-27 10:58:26.641 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3173a8c3822fa843,9e3e27d1277f498e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.40.198.13, traceId=3173a8c3822fa843, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2025-10-27 10:58:26.641 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3173a8c3822fa843,9e3e27d1277f498e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.40.198.13, traceId=3173a8c3822fa843, duration=4 ms] 2025-10-27 10:58:26.707 INFO 2458 --- [prometheus-minute-schedule-1,d9a877add8ee6356,70ff99f8493b101b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:26.715 INFO 2458 --- [prometheus-minute-schedule-1,d9a877add8ee6356,70ff99f8493b101b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:26.736 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,3b2b4feec05cbe28,6423419b41d80e45] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /favicon.ico?null, client=10.40.198.13, traceId=3b2b4feec05cbe28, method=void com.oceanbase.ocp.server.common.controller.web.WebController.dummyFavicon(), args=] 2025-10-27 10:58:26.737 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,3b2b4feec05cbe28,6423419b41d80e45] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /favicon.ico client=10.40.198.13, traceId=3b2b4feec05cbe28, duration=8 ms] 2025-10-27 10:58:26.809 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:26.810 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:27.211 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d0837be413688caa,2375c23868d0a11e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.40.198.13, traceId=d0837be413688caa, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-10-27 10:58:27.214 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d0837be413688caa,2375c23868d0a11e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.40.198.13, traceId=d0837be413688caa, duration=6 ms] 2025-10-27 10:58:27.411 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 10:58:27.411 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 10:58:27.411 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 10:58:28.440 WARN 2458 --- [ocp-async-4,8fd05246af60c2ae,edae53464fe8b718] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:28.440 INFO 2458 --- [ocp-async-3,1139b8602cd92169,579435acee1d4157] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:28.501 INFO 2458 --- [ocp-async-2,694fb2fb66822da6,302c4c952784a718] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:28.749 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 10:58:28.749 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 10:58:28.751 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 10:58:28.751 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839175233367121920 2025-10-27 10:58:28.752 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 10:58:28.752 INFO 2458 --- [ocp-schedules-5,0306f3a070f7fbde,cfa115f35ab3ad85] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 10:58:29.242 INFO 2458 --- [ocp-schedules-7,f8c29d56a3af7f87,225b3a78a8df6799] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 10:58:29.247 INFO 2458 --- [ocp-schedules-7,f8c29d56a3af7f87,225b3a78a8df6799] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 10:58:29.247 INFO 2458 --- [ocp-schedules-7,f8c29d56a3af7f87,225b3a78a8df6799] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 10:58:29.408 INFO 2458 --- [ocp-schedules-16,04e3c3b81a7cc8bd,fec178b027b221f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:58:29.472 WARN 2458 --- [scanner-112108723-1,04e3c3b81a7cc8bd,9a89c58fb5edb819] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:58:29.707 INFO 2458 --- [prometheus-minute-schedule-1,0091c74b7444dde2,d94e82ca7fd79a8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:29.719 INFO 2458 --- [prometheus-minute-schedule-1,0091c74b7444dde2,d94e82ca7fd79a8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:29.748 INFO 2458 --- [alarm-detect-14,04e3c3b81a7cc8bd,26661c8fa8aca661] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533880 ~ 1761533899 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533895, value=88.507)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533895, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533895=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:58:29.810 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:29.811 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:30.441 INFO 2458 --- [ocp-async-1,db889f964f1af94e,69001f405e345d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:30.441 WARN 2458 --- [ocp-async-9,5eb8ba8250c603cc,36922141c7c09919] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:31.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:31.718198+08:00, lastProcessTime=2025-10-27T10:58:21.714698+08:00 2025-10-27 10:58:31.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:31.718198+08:00 2025-10-27 10:58:32.441 INFO 2458 --- [ocp-async-7,d9903465b5225e6c,189041755569c8bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:32.441 WARN 2458 --- [ocp-async-10,f65efb4ae8935a83,47668d0e2a5d30c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:32.707 INFO 2458 --- [prometheus-minute-schedule-1,abfb8eb9ff8c308d,58494a35857430a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:32.716 INFO 2458 --- [prometheus-minute-schedule-1,abfb8eb9ff8c308d,58494a35857430a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:32.811 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:32.814 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:32.893 INFO 2458 --- [ocp-schedules-17,0c0f2670ae495d95,12dbcf8530ae9b11] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:32.897 INFO 2458 --- [ocp-schedules-17,0c0f2670ae495d95,12dbcf8530ae9b11] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:33.169 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 10:58:33.169 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 10:58:33.232 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:33.232 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:33.271 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:33.271 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:33.271 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:33.272 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 10:58:33.285 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:33.285 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:33.285 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:33.286 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 10:58:33.354 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:33.355 INFO 2458 --- [ocp-schedules-9,e1b6fa857dd43b9e,d026860951729006] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 10:58:33.658 INFO 2458 --- [ocp-schedules-13,e40ecee5d15da497,f324f81c1fd78b07] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 10:58:33.658 INFO 2458 --- [ocp-schedules-13,e40ecee5d15da497,f324f81c1fd78b07] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 10:58:34.441 INFO 2458 --- [ocp-async-4,5ffe4ef4b663584f,05455a91f4b4b940] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:34.441 WARN 2458 --- [ocp-async-3,f0ac23d3b6780baf,1e9922455762b127] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:35.426 INFO 2458 --- [ocp-schedules-26,64ed22549e43582d,c95bc938ef970557] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:35.707 INFO 2458 --- [prometheus-minute-schedule-1,6c4fb6ed513e676f,101eb5bb23507510] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:35.714 INFO 2458 --- [prometheus-minute-schedule-1,6c4fb6ed513e676f,101eb5bb23507510] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:35.814 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:35.815 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:36.026 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:36.026 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 10:58:36.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 10:58:36.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 10:58:36.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 10:58:36.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 10:58:36.040 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:36.040 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:36.042 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:36.044 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:36.044 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:36.045 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:36.047 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:36.047 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:36.050 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:36.050 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:36.098 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 10:58:36.101 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 10:58:36.103 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 10:58:36.105 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 10:58:36.109 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 10:58:36.110 INFO 2458 --- [ocp-schedules-17,009fedc9a106c062,bb544b180d891a4a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 10:58:36.441 WARN 2458 --- [ocp-async-9,3e92f85eeef2c17e,9d5e6ab9a24b90f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:36.441 INFO 2458 --- [ocp-async-2,afc7851eb25b1d64,e0dac504811d860f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:37.708 INFO 2458 --- [prometheus-cache-monitor1,fb0beabacc177a3a,8416b9909665ab04] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20264,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5072,"seriesIdKeySize":8767,"secondWriteQueue":0,"minuteWriteCacheSize":7850} 2025-10-27 10:58:37.708 INFO 2458 --- [prometheus-cache-monitor1,fb0beabacc177a3a,8416b9909665ab04] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 10:58:38.441 WARN 2458 --- [ocp-async-10,ec6e55714b73eaf8,f02d42295bba162d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:38.441 INFO 2458 --- [ocp-async-5,061015c8df05d10c,a5dc1994aa72baee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:38.501 INFO 2458 --- [ocp-async-3,2256db3411a1f9ca,b21d0077ab945c8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:38.527 INFO 2458 --- [ocp-schedules-27,efbb3b5ed6a8791f,cbcdd5180ade156c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 10:58:38.532 INFO 2458 --- [ocp-schedules-27,efbb3b5ed6a8791f,cbcdd5180ade156c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:38.532 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 10:58:38.532 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 10:58:38.533 INFO 2458 --- [ocp-schedules-27,efbb3b5ed6a8791f,cbcdd5180ade156c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:58:38.707 INFO 2458 --- [prometheus-minute-schedule-1,ac6dd7eafd645e25,a85d57501e175d35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:38.715 INFO 2458 --- [prometheus-minute-schedule-1,ac6dd7eafd645e25,a85d57501e175d35] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:38.815 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:38.816 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:39.759 INFO 2458 --- [ocp-schedules-12,90808568035eb509,d8425dc64174914d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:58:39.810 WARN 2458 --- [scanner-112108936-1,90808568035eb509,22a69a2b037bc934] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:58:40.013 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 10:58:40.013 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 10:58:40.038 INFO 2458 --- [ocp-schedules-27,0bda07649bec58c8,81c436b4fc7eedaf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 10:58:40.039 INFO 2458 --- [ocp-schedules-27,0bda07649bec58c8,81c436b4fc7eedaf] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 10:58:40.049 INFO 2458 --- [Check obproxy1,3b04e37be9d4cb5b,256f896143e19b25] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:58:40.049 INFO 2458 --- [Check obproxy1,3b04e37be9d4cb5b,256f896143e19b25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:58:40.053 INFO 2458 --- [Collect obproxy1,1e504da77939de5b,fd0897a3f0b0f4ee] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:58:40.053 INFO 2458 --- [Collect obproxy1,1e504da77939de5b,fd0897a3f0b0f4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:58:40.079 INFO 2458 --- [Check obproxy1,3b04e37be9d4cb5b,256f896143e19b25] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 10:58:40.082 INFO 2458 --- [Check obproxy1,3b04e37be9d4cb5b,256f896143e19b25] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 10:58:40.082 INFO 2458 --- [Check obproxy1,3b04e37be9d4cb5b,256f896143e19b25] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 10:58:40.082 INFO 2458 --- [Collect obproxy1,1e504da77939de5b,fd0897a3f0b0f4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 10:58:40.085 INFO 2458 --- [Collect obproxy1,1e504da77939de5b,fd0897a3f0b0f4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 10:58:40.088 INFO 2458 --- [Collect obproxy1,1e504da77939de5b,fd0897a3f0b0f4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 10:58:40.441 WARN 2458 --- [ocp-async-6,5ef0a69727c77f9f,d25c79b68378c156] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:40.441 INFO 2458 --- [ocp-async-6,3cf3882a178045e3,5e40b22479e78dc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:41.707 INFO 2458 --- [prometheus-minute-schedule-1,fcc146100596ab37,43ddf1cce8ca9609] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:41.714 INFO 2458 --- [prometheus-minute-schedule-1,fcc146100596ab37,43ddf1cce8ca9609] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:41.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:41.716694+08:00, lastProcessTime=2025-10-27T10:58:31.718198+08:00 2025-10-27 10:58:41.816 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:41.817 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:41.920 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:41.716694+08:00 2025-10-27 10:58:41.973 INFO 2458 --- [ocp-schedules-26,0d874e91f753ba89,a74a476f722a871a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 10:58:41.973 INFO 2458 --- [ocp-schedules-26,0d874e91f753ba89,a74a476f722a871a] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 10:58:41.981 INFO 2458 --- [ocp-schedules-26,0d874e91f753ba89,a74a476f722a871a] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:58:41.981 INFO 2458 --- [ocp-schedules-26,0d874e91f753ba89,a74a476f722a871a] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:58:42.442 WARN 2458 --- [ocp-async-9,406176930960c73b,cdf51597c4576ec9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:42.442 INFO 2458 --- [ocp-async-9,a47c90afdd9c0f53,bbfea42c2a140329] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:42.905 INFO 2458 --- [ocp-schedules-6,57bc5f4b509e0e93,01eae415ff0fd55f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:42.908 INFO 2458 --- [ocp-schedules-6,57bc5f4b509e0e93,01eae415ff0fd55f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:43.888 INFO 2458 --- [ocp-schedules-18,c6d15f74d0d897a1,4240024ed6f0c06b] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 10:58:44.442 WARN 2458 --- [ocp-async-10,4519ac2fc9e4468f,b943485e0909c2d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:44.442 INFO 2458 --- [ocp-async-10,1c94aa5a1df6e1ac,a459170290c053c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:44.707 INFO 2458 --- [prometheus-minute-schedule-1,f4de9a4bc0294ec5,d116a4080b73c2d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:44.717 INFO 2458 --- [prometheus-minute-schedule-1,f4de9a4bc0294ec5,d116a4080b73c2d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:44.817 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:44.818 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:45.102 INFO 2458 --- [prometheus-second-schedule-1,49af948a168d80b7,d6ab1ed2a3d06b5d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 10:58:45.433 INFO 2458 --- [ocp-schedules-19,25c60ec6cb0a7d6f,3c99a2a6e1686443] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:46.391 INFO 2458 --- [ocp-schedules-23,0313cc98a72d0213,61db7690309ab62f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:58:46.399 INFO 2458 --- [ocp-schedules-23,0313cc98a72d0213,61db7690309ab62f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:46.399 INFO 2458 --- [ocp-schedules-23,0313cc98a72d0213,61db7690309ab62f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:58:46.444 WARN 2458 --- [ocp-async-7,ae5f39d6a1b382de,50efc90befa56824] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:46.444 INFO 2458 --- [ocp-async-3,9de168fdf10977ee,64515254ff3efa1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:46.988 INFO 2458 --- [ocp-schedules-8,7aeaf8b853d72536,ae0911f7b69f2db7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T10:58:46.988+08:00, clusters=[1, 2] 2025-10-27 10:58:47.029 INFO 2458 --- [ocp-schedules-8,7aeaf8b853d72536,ae0911f7b69f2db7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 10:58:47.033 INFO 2458 --- [ocp-schedules-8,7aeaf8b853d72536,ae0911f7b69f2db7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 10:58:47.707 INFO 2458 --- [prometheus-minute-schedule-1,fa0e50374dffbce6,645fffff0e7635c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:47.723 INFO 2458 --- [prometheus-minute-schedule-1,fa0e50374dffbce6,645fffff0e7635c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:47.818 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:47.819 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:48.444 WARN 2458 --- [ocp-async-6,6b5c73fce285cf95,64ff1c277684378a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:48.444 INFO 2458 --- [ocp-async-2,2adc1cc2af3cd63b,80abd98deff26501] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:48.501 INFO 2458 --- [ocp-async-9,28f5f243f19cbdaa,c469eaa5105bf6b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:49.980 INFO 2458 --- [ocp-schedules-19,6fb8bea04a03214b,0100d7d4e58c9090] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:58:49.999 WARN 2458 --- [scanner-112109174-1,6fb8bea04a03214b,1147d6d8e48c8795] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:58:50.127 INFO 2458 --- [alarm-detect-8,6fb8bea04a03214b,142a3d828e7c8dd4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533895 ~ 1761533920 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533910, value=88.083)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533910, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533910=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:58:50.129 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,74f243d424f6c380,90404e5ce6d50e1a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=74f243d424f6c380, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 10:58:50.129 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,74f243d424f6c380,90404e5ce6d50e1a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=74f243d424f6c380, duration=1 ms] 2025-10-27 10:58:50.445 WARN 2458 --- [ocp-async-5,6fc2455f73f1a94a,176ab00c792652a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:50.445 INFO 2458 --- [ocp-async-1,a44e0edf9b0424c4,fe3546b01c335ac4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:50.707 INFO 2458 --- [prometheus-minute-schedule-1,104632154a4e7205,5a33b4946e9db95a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:50.714 INFO 2458 --- [prometheus-minute-schedule-1,104632154a4e7205,5a33b4946e9db95a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:50.819 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:50.820 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:51.409 INFO 2458 --- [ocp-schedules-8,5840092712c6f96b,0d1d9cd403ad5990] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:51.409 INFO 2458 --- [ocp-schedules-24,35ddbf3d9f52ead9,d1b40fccb098e349] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:58:51.410 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:58:51.411 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:58:51.411 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:58:51.411 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:58:51.425 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:51.425 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:51.426 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:51.426 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:51.428 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:51.428 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:51.428 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:51.429 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:51.430 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:51.430 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:51.430 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:51.430 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:51.431 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:51.431 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:51.431 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:51.432 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:51.432 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:51.433 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:51.435 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:51.435 INFO 2458 --- [ocp-schedules-26,baa8ec534218bf79,33bc664c4d7b5cd5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:58:51.435 INFO 2458 --- [host-check-clockdiff1,baa8ec534218bf79,c80d914543791bb3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 10:58:51.435 INFO 2458 --- [host-check-clockdiff2,baa8ec534218bf79,a0eb6557a35ce4ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 10:58:51.436 INFO 2458 --- [host-check-clockdiff4,baa8ec534218bf79,0b8bb616dfc01af2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 10:58:51.436 INFO 2458 --- [host-check-clockdiff3,baa8ec534218bf79,e33653b34044d722] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 10:58:51.436 INFO 2458 --- [host-check-clockdiff5,baa8ec534218bf79,8357595bf5e1b184] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 10:58:51.436 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:51.442 INFO 2458 --- [ocp-schedules-7,32c676fea31e81b2,16cfddeca5f2a9e8] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 10:58:51.476 INFO 2458 --- [host-check-clockdiff2,baa8ec534218bf79,a0eb6557a35ce4ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533931 -1 -1 , error= 2025-10-27 10:58:51.477 INFO 2458 --- [host-check-clockdiff4,baa8ec534218bf79,0b8bb616dfc01af2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533931 -1 -1 , error= 2025-10-27 10:58:51.502 INFO 2458 --- [host-check-clockdiff1,baa8ec534218bf79,c80d914543791bb3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533931 0 0 , error= 2025-10-27 10:58:51.507 INFO 2458 --- [host-check-clockdiff5,baa8ec534218bf79,8357595bf5e1b184] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533931 0 0 , error= 2025-10-27 10:58:51.546 INFO 2458 --- [ocp-schedules-28,550d32c06445b886,2d4de578f0d686d1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 10:58:51.550 INFO 2458 --- [ocp-schedules-9,ae70883f0ee23b80,d2f5101eb9618eeb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 10:58:51.550 INFO 2458 --- [ocp-schedules-28,550d32c06445b886,2d4de578f0d686d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:51.554 INFO 2458 --- [ocp-schedules-9,ae70883f0ee23b80,d2f5101eb9618eeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:51.554 INFO 2458 --- [ocp-schedules-29,d1378ac66d6e3308,39d32b985e819062] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 10:58:51.556 INFO 2458 --- [ocp-schedules-9,ae70883f0ee23b80,d2f5101eb9618eeb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 10:58:51.559 INFO 2458 --- [ocp-schedules-29,d1378ac66d6e3308,39d32b985e819062] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:51.568 INFO 2458 --- [ocp-schedules-23,95ede83424319054,ac9ebebbaf50970c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 10:58:51.569 INFO 2458 --- [ocp-schedules-12,7e6ba5d1565f4755,4e32698ae3529e9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 10:58:51.574 INFO 2458 --- [ocp-schedules-11,0a3e6747c0dffeae,d96a61318063d2ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 10:58:51.586 INFO 2458 --- [ocp-schedules-12,7e6ba5d1565f4755,4e32698ae3529e9e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 10:58:51.586 INFO 2458 --- [ocp-schedules-11,0a3e6747c0dffeae,d96a61318063d2ea] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 10:58:51.604 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 10:58:51.606 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T10:58:51.605978291+08:00 2025-10-27 10:58:51.610 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 10:58:51.612 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T10:58:51.606+08:00 2025-10-27 10:58:51.612 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T10:58:51.611911816+08:00 2025-10-27 10:58:51.613 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T10:58:51.612+08:00 2025-10-27 10:58:51.621 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T10:58:51.606+08:00 and status AVAILABLE 2025-10-27 10:58:51.625 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 10:58:51.626 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 10:58:51.626 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 10:58:51.627 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T10:58:51.612+08:00 and status AVAILABLE 2025-10-27 10:58:51.628 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 10:58:51.631 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 10:58:51.631 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 10:58:51.631 INFO 2458 --- [host-check-agent-status4,35ddbf3d9f52ead9,a1ff893772bc1136] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 10:58:51.628 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T10:58:51.626959101+08:00 2025-10-27 10:58:51.633 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T10:58:51.633030702+08:00 2025-10-27 10:58:51.633 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 10:58:51.634 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T10:58:51.633+08:00 2025-10-27 10:58:51.635 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T10:58:51.633+08:00 2025-10-27 10:58:51.636 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.637 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:51.638 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 10:58:51.642 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 10:58:51.644 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T10:58:51.633+08:00 and status AVAILABLE 2025-10-27 10:58:51.645 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 10:58:51.645 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 10:58:51.645 INFO 2458 --- [host-check-agent-status2,35ddbf3d9f52ead9,d96c156c90c1c406] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 10:58:51.647 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T10:58:51.633+08:00 and status AVAILABLE 2025-10-27 10:58:51.649 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 10:58:51.652 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 10:58:51.654 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 10:58:51.655 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 10:58:51.656 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 10:58:51.656 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 10:58:51.656 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T10:58:51.656861726+08:00 2025-10-27 10:58:51.657 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T10:58:51.657+08:00 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status3,35ddbf3d9f52ead9,8515d36a6a563081] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 10:58:51.659 INFO 2458 --- [host-check-agent-status1,35ddbf3d9f52ead9,523a105fc76066a6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 10:58:51.660 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 10:58:51.660 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 10:58:51.660 INFO 2458 --- [host-check-agent-status5,35ddbf3d9f52ead9,ab0904dadd378de7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 10:58:51.661 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:58:51.663 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:58:51.664 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:58:51.665 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.666 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:51.667 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T10:58:51.657+08:00 and status AVAILABLE 2025-10-27 10:58:51.667 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:58:51.670 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.670 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.671 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.671 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.671 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.672 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:51.672 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 10:58:51.676 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 10:58:51.676 INFO 2458 --- [ocp-schedules-28,550d32c06445b886,2d4de578f0d686d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:58:51.677 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:51.679 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:51.707 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:58:51.710 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:58:51.715 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:51.718 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:51.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:58:51.718653+08:00, lastProcessTime=2025-10-27T10:58:41.716694+08:00 2025-10-27 10:58:51.739 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.739 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.739 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.739 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:51.743 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:51.745 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:51.761 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 10:58:51.764 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 10:58:51.766 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:51.767 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:51.768 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:51.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:58:51.783 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:58:51.788 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 10:58:51.796 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 10:58:51.798 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 10:58:51.800 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 10:58:51.800 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 10:58:51.820 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 10:58:51.832 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 10:58:51.838 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:51.838 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:51.838 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:51.839 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.840 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:51.841 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.841 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:51.841 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.841 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:58:51.844 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:58:51.845 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:51.853 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 10:58:51.860 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:51.861 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 10:58:51.864 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 10:58:51.866 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:51.872 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:51.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:51.881 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 10:58:51.885 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 10:58:51.887 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 10:58:51.896 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 10:58:51.898 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:58:51.918 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:51.919 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:51.919 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 10:58:51.929 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 10:58:51.936 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 10:58:51.957 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:51.962 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 10:58:51.972 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:51.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.974 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:58:51.975 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 10:58:51.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:51.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:51.975 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:51.978 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 10:58:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:58:51.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:51.980 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:51.985 INFO 2458 --- [host-check-host-status4,5840092712c6f96b,a7af6de7001c2c55] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 10:58:51.988 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 10:58:51.988 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 10:58:51.993 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:51.994 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:58:51.718653+08:00 2025-10-27 10:58:51.998 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:58:52.000 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 10:58:52.001 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:58:52.006 INFO 2458 --- [host-check-host-status3,5840092712c6f96b,7148460f6cc4bb71] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 10:58:52.010 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 10:58:52.023 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:58:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:52.039 INFO 2458 --- [host-check-host-status5,5840092712c6f96b,a20fa86c61ea37c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 10:58:52.043 INFO 2458 --- [host-check-host-status2,5840092712c6f96b,c9c870dfbdabeb7d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 10:58:52.057 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 10:58:52.059 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:58:52.071 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.071 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:52.092 INFO 2458 --- [host-check-host-status1,5840092712c6f96b,a92ed9c4b0bb06b7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 10:58:52.103 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 10:58:52.103 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.104 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:58:52.106 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.106 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:58:52.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:52.126 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:58:52.137 INFO 2458 --- [ocp-schedules-1,7d24d531c15b5fdd,8552c6b4e50bce3e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 10:58:52.141 INFO 2458 --- [ocp-schedules-1,7d24d531c15b5fdd,8552c6b4e50bce3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:58:52.150 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:52.151 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.151 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:58:52.160 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.160 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:58:52.174 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:58:52.174 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 10:58:52.195 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.195 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:58:52.225 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:58:52.226 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.229 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:58:52.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:58:52.239 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.240 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.268 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.269 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.274 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.276 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 10:58:52.279 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:58:52.287 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 10:58:52.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:58:52.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:58:52.298 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.299 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.300 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.300 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.300 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.301 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.301 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.301 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.328 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.328 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.328 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.329 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.329 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.330 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:58:52.361 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.361 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.362 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.362 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.362 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.362 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:58:52.364 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.364 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.365 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.365 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.387 INFO 2458 --- [host-check-clockdiff3,baa8ec534218bf79,e33653b34044d722] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533932 -1 -1 , error= 2025-10-27 10:58:52.394 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.395 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.396 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.396 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.397 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.397 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 10:58:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.398 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:58:52.398 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.399 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.400 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.400 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:58:52.400 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.401 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.401 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.401 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.402 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.402 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.402 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.403 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.403 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 10:58:52.406 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:58:52.429 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.429 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.429 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.430 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:58:52.432 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.432 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.432 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.433 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:58:52.445 WARN 2458 --- [ocp-async-8,3aed9438069079f3,0ff210b8a945569c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:52.445 INFO 2458 --- [ocp-async-4,a023a05497d78056,aee0639b4b9f4b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:52.447 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:58:52.463 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:52.465 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.466 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:58:52.466 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.466 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.466 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.467 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:58:52.476 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.476 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 10:58:52.476 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.476 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 10:58:52.479 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 10:58:52.479 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:58:52.497 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.497 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.497 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.498 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.500 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:58:52.502 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.502 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.502 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.502 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:58:52.530 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.530 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.530 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.531 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:58:52.540 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:58:52.540 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533402507000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 10:58:52.540 INFO 2458 --- [ocp-schedules-29,d1378ac66d6e3308,39d32b985e819062] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:58:52.556 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.556 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.556 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.557 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:58:52.610 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.610 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.611 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.612 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.613 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.613 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.613 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.614 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.614 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.614 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.615 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.616 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.616 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.616 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.617 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.618 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.618 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.618 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.618 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.623 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.624 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.624 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.624 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.625 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.625 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.626 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.626 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.627 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.627 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.627 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.627 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.628 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.628 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.628 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.629 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.629 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 10:58:52.631 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.631 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.632 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:58:52.633 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.633 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.634 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:58:52.634 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 10:58:52.637 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:58:52.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:58:52.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:58:52.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:58:52.725 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:58:52.725 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:58:52.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 10:58:52.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 10:58:52.759 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:58:52.759 INFO 2458 --- [ocp-schedules-1,7d24d531c15b5fdd,8552c6b4e50bce3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 10:58:52.916 INFO 2458 --- [ocp-schedules-11,96266d70e8fa998c,c61dd4fb4cab8404] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:58:52.921 INFO 2458 --- [ocp-schedules-11,96266d70e8fa998c,c61dd4fb4cab8404] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:58:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 10:58:53.707 INFO 2458 --- [prometheus-minute-schedule-1,d82a0a6f8e6d9bce,28d7917f31261c3e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:53.715 INFO 2458 --- [prometheus-minute-schedule-1,d82a0a6f8e6d9bce,28d7917f31261c3e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:53.820 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:53.821 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:54.008 INFO 2458 --- [ocp-schedules-2,e4ef8df5fef8001f,208ed1e4fa1a5260] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 10:58:54.445 WARN 2458 --- [ocp-async-2,721d356b6b337c36,833148717b2a9b86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:54.445 INFO 2458 --- [ocp-async-9,387d2f9e3afba1da,36063ddb81c0e083] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:54.574 INFO 2458 --- [ocp-schedules-4,4f35ce8fa4e7506f,01d0cf0ea682a299] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-10-27 10:58:54.574 INFO 2458 --- [ocp-schedules-4,4f35ce8fa4e7506f,01d0cf0ea682a299] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-10-27 10:58:55.440 INFO 2458 --- [ocp-schedules-30,7606faaf3f90a8b9,f92c4dfcef5a30c1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:58:56.222 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,05b3ae928635a7ee,0f91f22b6fcf288f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=05b3ae928635a7ee, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 10:58:56.222 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,05b3ae928635a7ee,0f91f22b6fcf288f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=05b3ae928635a7ee, duration=10 ms] 2025-10-27 10:58:56.277 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,0bd8767149aa6171,2c08c4de8625fffe] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-10-27 10:58:56.436 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,0bd8767149aa6171,2c08c4de8625fffe] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-10-27 10:58:56.445 INFO 2458 --- [ocp-async-1,c84efdaa1a289508,d954856dc5d1e51c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:56.445 WARN 2458 --- [ocp-async-7,99988441a6614729,54e8a7a18ea2734f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:56.542 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,0bd8767149aa6171,2c08c4de8625fffe] .o.o.s.c.c.s.m.ClientLoginAttemptManager : Increase client ip login failed attempts, clientAddress=10.40.198.13, attempts=1 2025-10-27 10:58:56.543 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,0bd8767149aa6171,2c08c4de8625fffe] c.s.h.CustomAuthenticationFailureHandler : Authentication failure: ip=10.40.198.13, userId=100, username=admin, message=用户名或者密码错误，请重试。您还剩余4次机会, exceptionMsg=Bad credentials 2025-10-27 10:58:56.707 INFO 2458 --- [prometheus-minute-schedule-1,68384817530cd2d6,49ab2dac87fbbb5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:56.715 INFO 2458 --- [prometheus-minute-schedule-1,68384817530cd2d6,49ab2dac87fbbb5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:56.821 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:56.823 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:58:58.445 INFO 2458 --- [ocp-async-6,7c0b57b540f24cf2,8cf48a2cd5b8b5e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:58:58.445 WARN 2458 --- [ocp-async-8,def5514c1b23505c,90bc9e34a2eeec14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:58:58.501 INFO 2458 --- [ocp-async-4,eb83c2b8f9e9d1f8,602ed4f85faf6a3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:58:59.026 INFO 2458 --- [ocp-schedules-27,f54789197669f5c2,f3e672489615f14f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 10:58:59.707 INFO 2458 --- [prometheus-minute-schedule-1,94a013ce399d3ad8,fabc59d39a7a8eeb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761533940, interval=60 2025-10-27 10:58:59.715 INFO 2458 --- [prometheus-minute-schedule-1,94a013ce399d3ad8,fabc59d39a7a8eeb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:58:59.823 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:58:59.824 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:00.134 INFO 2458 --- [ocp-schedules-23,784651d34477d494,cf4e03aab0d298ee] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:00.192 WARN 2458 --- [scanner-112109368-1,784651d34477d494,85ea80d0f9a949fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:00.376 INFO 2458 --- [alarm-detect-9,784651d34477d494,61fcddfe53546cee] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533910 ~ 1761533930 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533925, value=63.282)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533925, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533925=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:59:00.445 WARN 2458 --- [ocp-async-5,da69a2490b64facb,08758323a8fb1b90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:00.445 INFO 2458 --- [ocp-async-2,be92a4cec3f4cd12,9ad0ebc9b03b440d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:01.350 INFO 2458 --- [ocp-schedules-19,7acd936019a3c1ea,930a33efc0d1f65b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 10:59:01.355 INFO 2458 --- [ocp-schedules-19,7acd936019a3c1ea,930a33efc0d1f65b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:01.398 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:01.399 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 10:59:01.416 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:01.416 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:01.416 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:01.416 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:01.417 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:01.417 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 10:59:01.435 INFO 2458 --- [ocp-schedules-19,7acd936019a3c1ea,930a33efc0d1f65b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 10:59:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:01.712961+08:00, lastProcessTime=2025-10-27T10:58:51.718653+08:00 2025-10-27 10:59:01.939 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:01.712961+08:00 2025-10-27 10:59:02.445 WARN 2458 --- [ocp-async-2,1ea9897b254c3804,9302a59f764f12b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:02.445 INFO 2458 --- [ocp-async-7,da9773004bba6f7d,48c9a7b320bb343e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:02.707 INFO 2458 --- [prometheus-minute-schedule-1,0d76fe4dc3ee2a78,65d412653d803916] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:02.716 INFO 2458 --- [prometheus-minute-schedule-1,0d76fe4dc3ee2a78,65d412653d803916] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:02.824 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:02.825 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:02.929 INFO 2458 --- [ocp-schedules-27,0e5f9af60d9c15b2,58b3f2cb6015c771] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:02.932 INFO 2458 --- [ocp-schedules-27,0e5f9af60d9c15b2,58b3f2cb6015c771] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:04.446 WARN 2458 --- [ocp-async-8,ed98bbad4622a810,d2a5a5c352ca37da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:04.446 INFO 2458 --- [ocp-async-3,6fa56880b873752a,8f277586bd901351] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:05.448 INFO 2458 --- [ocp-schedules-22,5a793b84ad1c2f0f,957f453e485904af] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:05.707 INFO 2458 --- [prometheus-minute-schedule-1,95ca9b059a78ebc6,8c82cee7e24b1623] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:05.715 INFO 2458 --- [prometheus-minute-schedule-1,95ca9b059a78ebc6,8c82cee7e24b1623] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:05.826 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:05.826 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:06.402 INFO 2458 --- [ocp-schedules-27,9572c52f5c2167b2,c55e02a8b4fed785] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:59:06.419 INFO 2458 --- [ocp-schedules-27,9572c52f5c2167b2,c55e02a8b4fed785] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:06.419 INFO 2458 --- [ocp-schedules-27,9572c52f5c2167b2,c55e02a8b4fed785] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:06.446 INFO 2458 --- [ocp-async-3,7a4a6cb6d11d02ad,b960cc317e06fb44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:06.446 WARN 2458 --- [ocp-async-9,0c2721ecbdec090d,bd7ee454bec545b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:07.099 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,0a23a0eac2400dbe,eb799f158157ea40] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=0a23a0eac2400dbe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 10:59:07.099 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,0a23a0eac2400dbe,eb799f158157ea40] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=0a23a0eac2400dbe, duration=0 ms] 2025-10-27 10:59:08.446 INFO 2458 --- [ocp-async-3,e5eb8b27e480cf94,fc62bff09cb00995] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:08.446 WARN 2458 --- [ocp-async-1,3e3e50e73ae629be,5aaf6517bec28da9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:08.502 INFO 2458 --- [ocp-async-8,79703dfb147dbd99,a61528028089a519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:08.707 INFO 2458 --- [prometheus-minute-schedule-1,352f45ed350e2217,2d025aaadc7b983d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:08.715 INFO 2458 --- [prometheus-minute-schedule-1,352f45ed350e2217,2d025aaadc7b983d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:08.826 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:08.827 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:10.015 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 10:59:10.015 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 10:59:10.097 INFO 2458 --- [Check obproxy1,ca61672c861c4f6e,150e06d6ed5a1ccd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:59:10.097 INFO 2458 --- [Check obproxy1,ca61672c861c4f6e,150e06d6ed5a1ccd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:59:10.097 INFO 2458 --- [Collect obproxy1,95620e37e1892b35,df9fb43be7eae356] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:59:10.097 INFO 2458 --- [Collect obproxy1,95620e37e1892b35,df9fb43be7eae356] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:59:10.127 INFO 2458 --- [Collect obproxy1,95620e37e1892b35,df9fb43be7eae356] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 10:59:10.128 INFO 2458 --- [Check obproxy1,ca61672c861c4f6e,150e06d6ed5a1ccd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 10:59:10.131 INFO 2458 --- [Collect obproxy1,95620e37e1892b35,df9fb43be7eae356] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 10:59:10.131 INFO 2458 --- [Check obproxy1,ca61672c861c4f6e,150e06d6ed5a1ccd] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 10:59:10.131 INFO 2458 --- [Check obproxy1,ca61672c861c4f6e,150e06d6ed5a1ccd] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 10:59:10.133 INFO 2458 --- [Collect obproxy1,95620e37e1892b35,df9fb43be7eae356] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 10:59:10.177 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 10:59:10.386 INFO 2458 --- [ocp-schedules-11,357dff64cce9bd7a,7591e83e510762d7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:10.418 WARN 2458 --- [scanner-112109600-1,357dff64cce9bd7a,abccef2b3d37557a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:10.446 INFO 2458 --- [ocp-async-8,12596bfb448605b7,60e520935f54417c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:10.447 WARN 2458 --- [ocp-async-6,c081aec2f6552ec5,f45e7cf50855c86d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:10.550 INFO 2458 --- [alarm-detect-14,357dff64cce9bd7a,156a0090b750e969] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533925 ~ 1761533940 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533940, value=67.956)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533940, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533940=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:59:11.707 INFO 2458 --- [prometheus-minute-schedule-1,db5203a5dd20051d,48360dfd81a17836] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:11.714 INFO 2458 --- [prometheus-minute-schedule-1,db5203a5dd20051d,48360dfd81a17836] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:11.714057+08:00, lastProcessTime=2025-10-27T10:59:01.712961+08:00 2025-10-27 10:59:11.827 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:11.828 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:11.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:11.714057+08:00 2025-10-27 10:59:12.125 INFO 2458 --- [ocp-schedules-12,2091409ac23df031,a5f395e2388b52a8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 10:59:12.125 INFO 2458 --- [ocp-schedules-12,2091409ac23df031,a5f395e2388b52a8] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 10:59:12.132 INFO 2458 --- [ocp-schedules-12,2091409ac23df031,a5f395e2388b52a8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:12.133 INFO 2458 --- [ocp-schedules-12,2091409ac23df031,a5f395e2388b52a8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:59:12.446 INFO 2458 --- [ocp-async-8,e31ec62c54a46769,ec90a56a4ec78bcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:12.446 WARN 2458 --- [ocp-async-10,474f5af253844955,d6dfee99dda69c6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:12.940 INFO 2458 --- [ocp-schedules-6,2fa4607e262c606a,dde720499914e106] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:12.944 INFO 2458 --- [ocp-schedules-6,2fa4607e262c606a,dde720499914e106] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:13.178 INFO 2458 --- [ocp-schedules-1,cf2492e8781b2192,2613414558789275] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 10:59:13.178 INFO 2458 --- [ocp-schedules-1,cf2492e8781b2192,2613414558789275] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 10:59:13.234 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 10:59:13.235 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 10:59:13.235 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 10:59:13.235 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 10:59:13.235 INFO 2458 --- [Collect all tenants major compactions6,cf2492e8781b2192,8ec870ee47281ba5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 10:59:13.235 INFO 2458 --- [Collect all tenants major compactions5,cf2492e8781b2192,d5f299ee1cc1d287] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 10:59:13.272 INFO 2458 --- [Collect all tenants major compactions6,cf2492e8781b2192,8ec870ee47281ba5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.273 INFO 2458 --- [Collect all tenants major compactions5,cf2492e8781b2192,d5f299ee1cc1d287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.274 INFO 2458 --- [Collect all tenants major compactions6,cf2492e8781b2192,8ec870ee47281ba5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 10:59:13.275 INFO 2458 --- [Collect all tenants major compactions5,cf2492e8781b2192,d5f299ee1cc1d287] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 10:59:13.277 INFO 2458 --- [Collect all tenants major compactions6,cf2492e8781b2192,8ec870ee47281ba5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 10:59:13.278 INFO 2458 --- [Collect all tenants major compactions5,cf2492e8781b2192,d5f299ee1cc1d287] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.287 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.288 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.288 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.288 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.289 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 10:59:13.289 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 10:59:13.290 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 10:59:13.292 INFO 2458 --- [Collect all tenants major compactions1,cf2492e8781b2192,18d22d3072e0c027] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 10:59:13.293 INFO 2458 --- [Collect all tenants major compactions3,cf2492e8781b2192,04aac36054db14d5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 10:59:13.293 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.293 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.294 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:13.294 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:13.294 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:13.295 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 10:59:13.295 INFO 2458 --- [Collect all tenants major compactions2,cf2492e8781b2192,b9cecc08b7f0f2de] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 10:59:13.298 INFO 2458 --- [Collect all tenants major compactions4,cf2492e8781b2192,0657b1643c0d979e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 10:59:13.299 INFO 2458 --- [ocp-schedules-1,cf2492e8781b2192,2613414558789275] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 10:59:13.378 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761533880, nodeCount=821, affectRows=821 2025-10-27 10:59:13.768 INFO 2458 --- [ocp-schedules-30,d01e18ec4619725e,72f07fb32a53228c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T10:59:13.768+08:00, clusters=[1, 2] 2025-10-27 10:59:13.770 INFO 2458 --- [ocp-schedules-30,d01e18ec4619725e,72f07fb32a53228c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 10:59:13.771 INFO 2458 --- [ocp-schedules-30,d01e18ec4619725e,72f07fb32a53228c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 10:59:13.812 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505643) Duplicate entry '835014129044889601-1761533820' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505643) Duplicate entry '835014129044889601-1761533820' for key 'PRIMARY' 2025-10-27 10:59:13.843 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=1000, affectRows=1000 2025-10-27 10:59:13.847 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=998, validCount=994 2025-10-27 10:59:13.853 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=1000, affectRows=1000 2025-10-27 10:59:13.853 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=1000, affectRows=1000 2025-10-27 10:59:13.858 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=995, validCount=993 2025-10-27 10:59:13.859 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=1000, validCount=1000 2025-10-27 10:59:13.860 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570098) Duplicate entry '835370602832797696-1761533820' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570098) Duplicate entry '835370602832797696-1761533820' for key 'PRIMARY' 2025-10-27 10:59:13.876 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=72, affectRows=72 2025-10-27 10:59:13.876 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=71, validCount=71 2025-10-27 10:59:14.446 WARN 2458 --- [ocp-async-2,dab95e8986619357,5c50a068d05a17ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:14.446 INFO 2458 --- [ocp-async-2,b524549fabfbdc5d,3b60134568cba5ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:14.707 INFO 2458 --- [prometheus-minute-schedule-1,ef9a9876ca692786,55cd9df577077ea8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:14.715 INFO 2458 --- [prometheus-minute-schedule-1,ef9a9876ca692786,55cd9df577077ea8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:14.828 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:14.829 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:15.456 INFO 2458 --- [ocp-schedules-17,06c1209f4e0ca360,adf77085c0c55940] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:16.446 WARN 2458 --- [ocp-async-6,767ad7daf5e2d243,6ccf049ceadbc1db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:16.447 INFO 2458 --- [ocp-async-5,bacf84a93afd1d84,d7356758a0f03081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:17.707 INFO 2458 --- [prometheus-minute-schedule-1,e0e096d708eeada6,2d54c291734ffdc6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:17.714 INFO 2458 --- [prometheus-minute-schedule-1,e0e096d708eeada6,2d54c291734ffdc6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:17.829 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:17.830 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:17.838 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=1000, affectRows=1000 2025-10-27 10:59:17.842 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=996, validCount=990 2025-10-27 10:59:18.176 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533880, nodeCount=1000, affectRows=1000 2025-10-27 10:59:18.179 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533880, nodeCount=996, validCount=981 2025-10-27 10:59:18.447 WARN 2458 --- [ocp-async-10,156131ef9c69a774,489c175218ce5e75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:18.447 INFO 2458 --- [ocp-async-4,76788fa8ea7a7c31,6ade8e1f043fc861] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:18.502 INFO 2458 --- [ocp-async-7,b8e0d134746bd737,b317b772b0833df8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:18.694 INFO 2458 --- [ocp-schedules-3,778b011c43ea06d5,2db9e5e0ddb52bb9] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 10:59:20.447 WARN 2458 --- [ocp-async-7,9866e91056f517dd,52bb720e4a86f904] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:20.447 INFO 2458 --- [ocp-async-2,a6b11b50e5ebcdea,5fc7891b9ea2a8c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:20.562 INFO 2458 --- [ocp-schedules-7,39241efbed7b6708,56a8ed23790f6348] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:20.595 WARN 2458 --- [scanner-112111339-1,39241efbed7b6708,57d77431be9d2cf2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:20.707 INFO 2458 --- [prometheus-minute-schedule-1,158b904f0db882fb,1cb743d0393974b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:20.715 INFO 2458 --- [prometheus-minute-schedule-1,158b904f0db882fb,1cb743d0393974b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:20.830 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:20.831 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:21.409 INFO 2458 --- [ocp-schedules-25,04c89b4c5c24ce87,cdf9585cf8d33953] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:21.409 INFO 2458 --- [ocp-schedules-5,88ad1ad1e86e5bea,d3b167cd5d480d57] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:21.410 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:59:21.410 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:59:21.411 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:59:21.411 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:59:21.411 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:59:21.411 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:59:21.412 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:59:21.412 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:59:21.412 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:59:21.412 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:59:21.424 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:21.424 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:21.424 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:21.424 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:21.424 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:21.425 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:21.426 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:21.429 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:21.429 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:21.429 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:21.429 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:21.430 INFO 2458 --- [ocp-schedules-28,756231733c574a6f,75542a1066a941ad] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-clockdiff1,756231733c574a6f,25bcac786c6aab7f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-clockdiff2,756231733c574a6f,8a13bc92db1ff123] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 10:59:21.430 INFO 2458 --- [host-check-clockdiff3,756231733c574a6f,8fef5f75a0ffd5d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 10:59:21.431 INFO 2458 --- [host-check-clockdiff4,756231733c574a6f,bfd6bef9992f44b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 10:59:21.431 INFO 2458 --- [host-check-clockdiff5,756231733c574a6f,293939f42830a54b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 10:59:21.431 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:21.431 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:21.432 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:21.432 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:21.434 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:21.436 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:21.449 INFO 2458 --- [ocp-schedules-23,94be596cbedbf080,49cd8d6b2e4e48b9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 10:59:21.449 INFO 2458 --- [host-check-clockdiff4,756231733c574a6f,bfd6bef9992f44b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533961 -1 -1 , error= 2025-10-27 10:59:21.449 INFO 2458 --- [host-check-clockdiff3,756231733c574a6f,8fef5f75a0ffd5d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533961 -1 -1 , error= 2025-10-27 10:59:21.498 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 10:59:21.499 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 10:59:21.499 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 10:59:21.500 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 10:59:21.501 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T10:59:21.500129521+08:00 2025-10-27 10:59:21.501 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T10:59:21.501554591+08:00 2025-10-27 10:59:21.501 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T10:59:21.501765773+08:00 2025-10-27 10:59:21.501 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 10:59:21.502 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T10:59:21.499949441+08:00 2025-10-27 10:59:21.502 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T10:59:21.501+08:00 2025-10-27 10:59:21.502 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T10:59:21.501+08:00 2025-10-27 10:59:21.502 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T10:59:21.502+08:00 2025-10-27 10:59:21.502 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T10:59:21.501+08:00 2025-10-27 10:59:21.503 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 10:59:21.504 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T10:59:21.502713114+08:00 2025-10-27 10:59:21.504 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 10:59:21.504 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 10:59:21.505 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T10:59:21.504+08:00 2025-10-27 10:59:21.506 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 10:59:21.506 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 10:59:21.506 INFO 2458 --- [host-check-agent-status2,88ad1ad1e86e5bea,8dae1914fd560022] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 10:59:21.507 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 10:59:21.507 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 10:59:21.507 INFO 2458 --- [host-check-agent-status3,88ad1ad1e86e5bea,31b21a14574dbb21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 10:59:21.508 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 10:59:21.508 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 10:59:21.508 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 10:59:21.508 INFO 2458 --- [host-check-agent-status1,88ad1ad1e86e5bea,672a21aac846ac1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 10:59:21.509 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 10:59:21.510 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T10:59:21.501+08:00 and status AVAILABLE 2025-10-27 10:59:21.510 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 10:59:21.510 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 10:59:21.510 INFO 2458 --- [host-check-agent-status5,88ad1ad1e86e5bea,0d2adf67ddde42df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 10:59:21.510 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T10:59:21.502+08:00 and status AVAILABLE 2025-10-27 10:59:21.511 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T10:59:21.501+08:00 and status AVAILABLE 2025-10-27 10:59:21.511 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 10:59:21.511 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T10:59:21.501+08:00 and status AVAILABLE 2025-10-27 10:59:21.511 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 10:59:21.511 INFO 2458 --- [host-check-agent-status4,88ad1ad1e86e5bea,49c0b5f4f3e176b1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 10:59:21.512 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T10:59:21.504+08:00 and status AVAILABLE 2025-10-27 10:59:21.514 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 10:59:21.516 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 10:59:21.516 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 10:59:21.516 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 10:59:21.518 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 10:59:21.533 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:21.534 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:21.537 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:21.537 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:21.538 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:21.538 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:21.540 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:21.541 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:21.542 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:21.545 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:21.546 INFO 2458 --- [ocp-schedules-9,fd7870fe816a2802,93d7ab5ee290c8b7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 10:59:21.549 INFO 2458 --- [ocp-schedules-24,0b9029cb03830e01,6fb99b73f1d347a9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 10:59:21.550 INFO 2458 --- [ocp-schedules-9,fd7870fe816a2802,93d7ab5ee290c8b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:21.553 INFO 2458 --- [ocp-schedules-24,0b9029cb03830e01,6fb99b73f1d347a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:21.554 INFO 2458 --- [ocp-schedules-3,c628bad329295b4b,d19c2bae4b781d3e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 10:59:21.554 INFO 2458 --- [ocp-schedules-24,0b9029cb03830e01,6fb99b73f1d347a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 10:59:21.557 INFO 2458 --- [ocp-schedules-3,c628bad329295b4b,d19c2bae4b781d3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:21.570 INFO 2458 --- [ocp-schedules-21,67fea598421d79fe,02b69d235592aa99] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 10:59:21.602 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:21.602 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.602 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:21.603 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:21.604 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 10:59:21.605 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:21.611 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:21.618 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.618 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.619 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 10:59:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:21.621 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 10:59:21.622 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 10:59:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 10:59:21.627 INFO 2458 --- [ocp-schedules-9,fd7870fe816a2802,93d7ab5ee290c8b7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:59:21.633 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 10:59:21.641 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.643 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:21.643 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 10:59:21.663 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 10:59:21.663 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 10:59:21.663 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 10:59:21.669 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 10:59:21.686 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 10:59:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.692 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.692 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:21.695 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:21.699 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:21.701 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:21.701 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:21.701 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:21.702 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.702 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:59:21.713 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 10:59:21.714 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 10:59:21.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:21.713685+08:00, lastProcessTime=2025-10-27T10:59:11.714057+08:00 2025-10-27 10:59:21.721 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:21.723 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 10:59:21.726 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 10:59:21.747 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 10:59:21.747 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 10:59:21.756 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 10:59:21.759 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 10:59:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:21.764 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 10:59:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:21.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.770 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:59:21.770 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:21.784 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.787 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:59:21.796 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 10:59:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:21.801 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 10:59:21.802 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 10:59:21.805 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:21.808 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:21.824 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 10:59:21.825 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:21.829 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 10:59:21.839 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:21.839 INFO 2458 --- [host-check-host-status5,04c89b4c5c24ce87,b4c1c584ba0e3445] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 10:59:21.845 INFO 2458 --- [host-check-host-status4,04c89b4c5c24ce87,b40d0831058bf6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 10:59:21.858 INFO 2458 --- [host-check-host-status2,04c89b4c5c24ce87,d8d85ef5cce47401] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 10:59:21.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:21.861 INFO 2458 --- [host-check-host-status3,04c89b4c5c24ce87,8a412aed77600453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 10:59:21.861 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.862 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:21.863 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 10:59:21.865 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1f71b335e0dbeab4,5d43c0984d1e8bcd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=1f71b335e0dbeab4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 10:59:21.865 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1f71b335e0dbeab4,5d43c0984d1e8bcd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=1f71b335e0dbeab4, duration=6 ms] 2025-10-27 10:59:21.870 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 10:59:21.875 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 10:59:21.895 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:21.895 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.896 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.896 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.896 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.896 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:21.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.897 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:21.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:21.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:21.898 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:21.900 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:21.901 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:59:21.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:21.916 INFO 2458 --- [host-check-host-status1,04c89b4c5c24ce87,cdff4a879a9bfd43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 10:59:21.919 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,bc98d0f9d0b2485b,0b88fd686c4ab0be] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-10-27 10:59:21.949 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:21.969 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:59:21.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:21.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:22.003 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:21.713685+08:00 2025-10-27 10:59:22.011 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 10:59:22.012 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.013 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:59:22.014 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.016 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:59:22.027 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,bc98d0f9d0b2485b,0b88fd686c4ab0be] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-10-27 10:59:22.040 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:59:22.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:22.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:22.064 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:22.064 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.065 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:59:22.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:22.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:22.095 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:59:22.095 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 10:59:22.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:22.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:22.131 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,bc98d0f9d0b2485b,0b88fd686c4ab0be] .o.o.s.c.c.s.m.ClientLoginAttemptManager : Increase client ip login failed attempts, clientAddress=10.40.198.13, attempts=2 2025-10-27 10:59:22.131 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,bc98d0f9d0b2485b,0b88fd686c4ab0be] c.s.h.CustomAuthenticationFailureHandler : Authentication failure: ip=10.40.198.13, userId=100, username=admin, message=用户名或者密码错误，请重试。您还剩余3次机会, exceptionMsg=Bad credentials 2025-10-27 10:59:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:22.204 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 10:59:22.215 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 10:59:22.224 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:59:22.280 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 10:59:22.280 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 10:59:22.283 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 10:59:22.283 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 10:59:22.283 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 10:59:22.283 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 10:59:22.286 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 10:59:22.286 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 10:59:22.286 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 10:59:22.286 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 10:59:22.308 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 10:59:22.313 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 10:59:22.313 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 10:59:22.313 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 10:59:22.313 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 10:59:22.313 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 10:59:22.319 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 10:59:22.329 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 10:59:22.329 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.329 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.329 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.330 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:59:22.332 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.332 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.332 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.333 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:59:22.378 INFO 2458 --- [host-check-clockdiff2,756231733c574a6f,8a13bc92db1ff123] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533962 -1 -1 , error= 2025-10-27 10:59:22.381 INFO 2458 --- [host-check-clockdiff1,756231733c574a6f,25bcac786c6aab7f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533962 0 0 , error= 2025-10-27 10:59:22.392 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:59:22.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:22.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:22.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:22.432 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:22.433 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:59:22.447 WARN 2458 --- [ocp-async-6,b253193c6b1bf2ef,bcaf2ab0d1468b8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:22.447 INFO 2458 --- [ocp-async-6,1c0a7f309d876e59,7f45c52ff7446ec8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:22.482 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:59:22.482 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533402507000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 10:59:22.482 INFO 2458 --- [ocp-schedules-3,c628bad329295b4b,d19c2bae4b781d3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:59:22.703 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 10:59:22.952 INFO 2458 --- [ocp-schedules-14,be181b56db249a3f,20ca5f835a2a6449] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:22.955 INFO 2458 --- [ocp-schedules-14,be181b56db249a3f,20ca5f835a2a6449] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:23.087 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.800321865S 2025-10-27 10:59:23.087 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 10:59:23.151 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 10:59:23.151 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 10:59:23.182 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 10:59:23.231 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 10:59:23.259 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 10:59:23.382 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] o.s.boot.SpringApplication : Started application in 1.099 seconds (JVM running for 256431.809) 2025-10-27 10:59:23.382 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.099374487S 2025-10-27 10:59:23.706 INFO 2458 --- [ocp-schedules-1,cb098253ea13f8da,096217ed71c42cba] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 10:59:23.707 INFO 2458 --- [prometheus-minute-schedule-1,67d0d8ada6f08fc5,e783fa3a91295b03] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:23.714 INFO 2458 --- [prometheus-minute-schedule-1,67d0d8ada6f08fc5,e783fa3a91295b03] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:23.831 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:23.831 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:23.966 INFO 2458 --- [host-check-clockdiff5,756231733c574a6f,293939f42830a54b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533963 1 1 , error= 2025-10-27 10:59:24.447 WARN 2458 --- [ocp-async-9,3bedd85cbd00cb8b,c170eb12ab3e22d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:24.447 INFO 2458 --- [ocp-async-9,8ff9aaa4144387ae,9860a6ed2c866dc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:25.462 INFO 2458 --- [ocp-schedules-3,c368f50ba9f86d7c,a6c9a108ef5ec778] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:26.422 INFO 2458 --- [ocp-schedules-10,2ab3ff03700b653e,4d16bb6203f8430d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:59:26.429 INFO 2458 --- [ocp-schedules-10,2ab3ff03700b653e,4d16bb6203f8430d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:26.429 INFO 2458 --- [ocp-schedules-10,2ab3ff03700b653e,4d16bb6203f8430d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:26.447 WARN 2458 --- [ocp-async-4,782c9051572c75b1,fde4651c98e350c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:26.447 INFO 2458 --- [ocp-async-7,93722a6838527874,30d73e6a99e7484c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:26.707 INFO 2458 --- [prometheus-minute-schedule-1,04b283485ed69dd4,3521e3782dc517da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:26.715 INFO 2458 --- [prometheus-minute-schedule-1,04b283485ed69dd4,3521e3782dc517da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:26.831 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:26.832 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:28.447 WARN 2458 --- [ocp-async-2,1a28b4b52b32c287,58bd55da7008f230] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:28.447 INFO 2458 --- [ocp-async-2,03e0200618152de3,a0d52ba21a0bf7a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:28.502 INFO 2458 --- [ocp-async-8,9fcc6aab32a4c58c,d2331aea9b9c6e5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:28.754 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 10:59:28.754 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 10:59:28.761 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 10:59:28.761 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839175233367121920 2025-10-27 10:59:28.765 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=839256866908741632 2025-10-27 10:59:28.766 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 10:59:28.766 INFO 2458 --- [ocp-schedules-10,42719260695219db,3a0b6b5d8e814360] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 10:59:29.247 INFO 2458 --- [ocp-schedules-2,f81688ed3982bad0,9f6d91bab4761e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 10:59:29.259 INFO 2458 --- [ocp-schedules-2,f81688ed3982bad0,9f6d91bab4761e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 10:59:29.259 INFO 2458 --- [ocp-schedules-2,f81688ed3982bad0,9f6d91bab4761e6c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 10:59:29.707 INFO 2458 --- [prometheus-minute-schedule-1,cc1444f196dde217,6a7401be6bc336c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:29.715 INFO 2458 --- [prometheus-minute-schedule-1,cc1444f196dde217,6a7401be6bc336c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:29.832 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:29.833 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:30.447 WARN 2458 --- [ocp-async-10,02f60b24599e5179,d262c642fe6b0d92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:30.448 INFO 2458 --- [ocp-async-9,a7bbdbc8ef9afd4b,5b4e4312f0b6904d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:30.576 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fac3228859915627,b74b58ebf3779a1d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=fac3228859915627, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 10:59:30.576 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fac3228859915627,b74b58ebf3779a1d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=fac3228859915627, duration=7 ms] 2025-10-27 10:59:30.622 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,065f9ab670a360d2,396a62f857565822] c.o.o.s.c.c.s.s.JdbcUserDetailsService : user 100 login with organization 10000000 2025-10-27 10:59:30.665 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,065f9ab670a360d2,396a62f857565822] c.c.s.h.OcpAuthenticationSuccessListener : User[name=admin, origin=LOCAL] authenticated successes. 2025-10-27 10:59:30.682 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,065f9ab670a360d2,396a62f857565822] c.s.h.CustomAuthenticationSuccessHandler : Authentication successful for /api/v2/iam/login 2025-10-27 10:59:30.718 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,065f9ab670a360d2,396a62f857565822] c.s.h.CustomAuthenticationSuccessHandler : Redirecting to saved url: / 2025-10-27 10:59:30.774 INFO 2458 --- [ocp-schedules-7,e1a8f43e00bc746c,539f5ebaea850cb3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:30.798 WARN 2458 --- [scanner-112111595-1,e1a8f43e00bc746c,1bd1c6e4841511fc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:31.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,b984ee323924e85a,95a29d38804d306a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=b984ee323924e85a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 10:59:31.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,a18a10d17479f0d5,9ceedfd1061a6bf1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=a18a10d17479f0d5, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 10:59:31.223 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7a14b92cbf179d2b,8ac4137ede27e34c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=7a14b92cbf179d2b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 10:59:31.223 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7a14b92cbf179d2b,8ac4137ede27e34c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=7a14b92cbf179d2b, duration=17 ms] 2025-10-27 10:59:31.236 INFO 2458 --- [alarm-detect-8,e1a8f43e00bc746c,a5dc094b9448a903] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533940 ~ 1761533961 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533955, value=76.904)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533955, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533955=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:59:31.265 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,b984ee323924e85a,95a29d38804d306a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=b984ee323924e85a, duration=59 ms] 2025-10-27 10:59:31.267 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,a18a10d17479f0d5,9ceedfd1061a6bf1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=a18a10d17479f0d5, duration=61 ms] 2025-10-27 10:59:31.352 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,57a907980d8c426a,79971315feb67058] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=57a907980d8c426a, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 10:59:31.412 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,04c27ec6dcb21f97,7a050acc401cb237] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=04c27ec6dcb21f97, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 10:59:31.424 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 10:59:31.424 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 10:59:31.424 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 10:59:31.426 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,57a907980d8c426a,79971315feb67058] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=57a907980d8c426a, duration=88 ms] 2025-10-27 10:59:31.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:31.718793+08:00, lastProcessTime=2025-10-27T10:59:21.713685+08:00 2025-10-27 10:59:31.776 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,04c27ec6dcb21f97,7a050acc401cb237] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=04c27ec6dcb21f97, duration=380 ms] 2025-10-27 10:59:31.782 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,de9f914ccab76c5d,8ad4dbc19cae8061] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=cluster%2Ctenant%2Cobproxy%2Chost%2Calarm%2Ctask, client=10.40.198.13, traceId=de9f914ccab76c5d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[cluster, tenant, obproxy, host, alarm, task],] 2025-10-27 10:59:31.783 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,0fb2110eb7803cf1,48bdea91befe88b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=7, client=10.40.198.13, traceId=0fb2110eb7803cf1, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-10-27 10:59:31.788 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,551a28a524470c81,897fd2157320dca1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.40.198.13, traceId=551a28a524470c81, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-10-27 10:59:31.788 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,6e27fdfbd16da72a,2cdf0a3def5202ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=backup, client=10.40.198.13, traceId=6e27fdfbd16da72a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[backup],] 2025-10-27 10:59:31.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,6e51470b84b2ab29,9abd3cad94f010ab] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/resourceStats?modules=cluster, client=10.40.198.13, traceId=6e51470b84b2ab29, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.resourceStatistic(List), args=[cluster],] 2025-10-27 10:59:31.822 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,6e51470b84b2ab29,9abd3cad94f010ab] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/resourceStats client=10.40.198.13, traceId=6e51470b84b2ab29, duration=48 ms] 2025-10-27 10:59:31.828 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,de9f914ccab76c5d,8ad4dbc19cae8061] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.40.198.13, traceId=de9f914ccab76c5d, duration=54 ms] 2025-10-27 10:59:31.828 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,6e27fdfbd16da72a,2cdf0a3def5202ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.40.198.13, traceId=6e27fdfbd16da72a, duration=54 ms] 2025-10-27 10:59:31.843 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,0fb2110eb7803cf1,48bdea91befe88b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.40.198.13, traceId=0fb2110eb7803cf1, duration=69 ms] 2025-10-27 10:59:31.864 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,130dc578320a2d7e,8c59118e93650972] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/status?modules=performance, client=10.40.198.13, traceId=130dc578320a2d7e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewStatus(List), args=[performance],] 2025-10-27 10:59:31.864 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,80e8f7fdc4521e56,9a758a1be5d28aba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_TENANT&type=TOP, client=10.40.198.13, traceId=80e8f7fdc4521e56, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_TENANT,OBCLUSTER,] 2025-10-27 10:59:31.864 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,130dc578320a2d7e,8c59118e93650972] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/status client=10.40.198.13, traceId=130dc578320a2d7e, duration=6 ms] 2025-10-27 10:59:31.877 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,f9ebd0d8f77736dc,f0369b1f8cbf9562] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_cpu_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.40.198.13, traceId=f9ebd0d8f77736dc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_cpu_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-10-27 10:59:31.882 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1aeeedfb620bcc7a,ab75af301153d55e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_ODP&type=TOP, client=10.40.198.13, traceId=1aeeedfb620bcc7a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_ODP,OBCLUSTER,] 2025-10-27 10:59:31.883 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,a1fec67f267ca1c1,4fc2039a112d0594] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=OVERVIEW_HOST&type=TOP, client=10.40.198.13, traceId=a1fec67f267ca1c1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,OVERVIEW_HOST,OBCLUSTER,] 2025-10-27 10:59:31.925 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,551a28a524470c81,897fd2157320dca1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.40.198.13, traceId=551a28a524470c81, duration=151 ms] 2025-10-27 10:59:31.934 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-3,f9ebd0d8f77736dc,f0369b1f8cbf9562] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = ob_cpu_assigned_percentage 2025-10-27 10:59:31.934 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-3,f9ebd0d8f77736dc,f0369b1f8cbf9562] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = ob_cpu_assigned_percentage, minTimestamp = 0, minTimeBuffer = 0, startTime = 1761533400 2025-10-27 10:59:31.939 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,a1fec67f267ca1c1,4fc2039a112d0594] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=a1fec67f267ca1c1, duration=69 ms] 2025-10-27 10:59:31.939 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1aeeedfb620bcc7a,ab75af301153d55e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=1aeeedfb620bcc7a, duration=69 ms] 2025-10-27 10:59:31.940 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,80e8f7fdc4521e56,9a758a1be5d28aba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=80e8f7fdc4521e56, duration=82 ms] 2025-10-27 10:59:31.945 WARN 2458 --- [scanner-112111743-1,f9ebd0d8f77736dc,6164372cae34daea] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_server_resource_cpu_assigned, seriesId size = 4 2025-10-27 10:59:31.945 WARN 2458 --- [scanner-112111744-1,f9ebd0d8f77736dc,957b31b02ff53196] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_server_resource_cpu, seriesId size = 4 2025-10-27 10:59:31.945 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,f9ebd0d8f77736dc,f0369b1f8cbf9562] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.40.198.13, traceId=f9ebd0d8f77736dc, duration=75 ms] 2025-10-27 10:59:31.956 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,0dbff6c52a50a9f9,887398f7e6269ad1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_disk_used_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.40.198.13, traceId=0dbff6c52a50a9f9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_disk_used_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-10-27 10:59:31.976 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,0dbff6c52a50a9f9,887398f7e6269ad1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = ob_disk_used_percentage 2025-10-27 10:59:31.976 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,0dbff6c52a50a9f9,887398f7e6269ad1] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = ob_disk_used_percentage, minTimestamp = 0, minTimeBuffer = 0, startTime = 1761533400 2025-10-27 10:59:31.977 WARN 2458 --- [scanner-112111745-1,0dbff6c52a50a9f9,986604d582620d9b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_disk_free_bytes, seriesId size = 4 2025-10-27 10:59:31.978 WARN 2458 --- [scanner-112111746-1,0dbff6c52a50a9f9,9b8db871e1dde52d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_disk_total_bytes, seriesId size = 4 2025-10-27 10:59:31.978 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,0dbff6c52a50a9f9,887398f7e6269ad1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.40.198.13, traceId=0dbff6c52a50a9f9, duration=40 ms] 2025-10-27 10:59:32.013 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5b932960229ffb00,83397d28ae9a4010] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/cluster/resources?metrics=ob_memory_assigned_percentage&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id, client=10.40.198.13, traceId=5b932960229ffb00, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewClusterResources(String,String,String,Integer), args=ob_memory_assigned_percentage,app,ob_cluster_name,ob_cluster_id,app:OB,5,] 2025-10-27 10:59:32.014 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,338dcebdf4ffbb27,dd98f1231022d327] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_tenant_disk_usage_v4&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.40.198.13, traceId=338dcebdf4ffbb27, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_tenant_disk_usage_v4,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-10-27 10:59:32.015 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,ea9c0db7d6b6621b,0b0cecc522316dc5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_cpu_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.40.198.13, traceId=ea9c0db7d6b6621b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_cpu_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-10-27 10:59:32.015 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,944176eae1fab1d1,4782f7f3c14a143b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?foldStandby=false, client=10.40.198.13, traceId=944176eae1fab1d1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-10-27 10:59:32.021 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c0494a851e1d144a,0e9974afea142100] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/overview/tenant/resources?metrics=ob_memory_percent&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name, client=10.40.198.13, traceId=c0494a851e1d144a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.OcpOverviewController.getOverviewTenantResources(String,String,String,Integer), args=ob_memory_percent,app,ob_cluster_name,ob_cluster_id,tenant_name,app:OB,5,] 2025-10-27 10:59:32.033 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-8,ea9c0db7d6b6621b,0b0cecc522316dc5] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async cannot find recordId. metric = ob_cpu_percent, labels = , groupBy = app,ob_cluster_name,ob_cluster_id,tenant_name, interval = 60 2025-10-27 10:59:32.035 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,ea9c0db7d6b6621b,0b0cecc522316dc5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.40.198.13, traceId=ea9c0db7d6b6621b, duration=28 ms] 2025-10-27 10:59:32.046 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:31.718793+08:00 2025-10-27 10:59:32.054 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-10,5b932960229ffb00,83397d28ae9a4010] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = ob_memory_assigned_percentage 2025-10-27 10:59:32.054 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-10,5b932960229ffb00,83397d28ae9a4010] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = ob_memory_assigned_percentage, minTimestamp = 0, minTimeBuffer = 0, startTime = 1761533400 2025-10-27 10:59:32.055 WARN 2458 --- [scanner-112111749-1,5b932960229ffb00,e5716b2ca7c00af8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_server_resource_memory_assigned_bytes, seriesId size = 4 2025-10-27 10:59:32.055 WARN 2458 --- [scanner-112111750-1,5b932960229ffb00,05d2d7f43072dd08] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_server_resource_memory_bytes, seriesId size = 4 2025-10-27 10:59:32.056 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5b932960229ffb00,83397d28ae9a4010] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/cluster/resources client=10.40.198.13, traceId=5b932960229ffb00, duration=50 ms] 2025-10-27 10:59:32.060 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,338dcebdf4ffbb27,dd98f1231022d327] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = ob_tenant_disk_usage_v4 2025-10-27 10:59:32.060 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,338dcebdf4ffbb27,dd98f1231022d327] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = ob_tenant_disk_usage_v4, minTimestamp = 0, minTimeBuffer = 0, startTime = 1761533400 2025-10-27 10:59:32.060 WARN 2458 --- [scanner-112111751-1,338dcebdf4ffbb27,e3679c2aa3c70321] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = ob_tenant_server_required_size, seriesId size = 14 2025-10-27 10:59:32.061 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,338dcebdf4ffbb27,dd98f1231022d327] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.40.198.13, traceId=338dcebdf4ffbb27, duration=54 ms] 2025-10-27 10:59:32.065 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-6,c0494a851e1d144a,0e9974afea142100] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async get empty result, metric = ob_memory_percent 2025-10-27 10:59:32.066 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-6,c0494a851e1d144a,0e9974afea142100] c.o.o.m.s.MonitorAsyncComputeServiceImpl : Scan data async result range is away from start time, fall back to full query. metric = ob_memory_percent, minTimestamp = 0, minTimeBuffer = 0, startTime = 1761533400 2025-10-27 10:59:32.068 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c0494a851e1d144a,0e9974afea142100] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/overview/tenant/resources client=10.40.198.13, traceId=c0494a851e1d144a, duration=62 ms] 2025-10-27 10:59:32.069 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,944176eae1fab1d1,4782f7f3c14a143b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.40.198.13, traceId=944176eae1fab1d1, duration=63 ms] 2025-10-27 10:59:32.448 INFO 2458 --- [ocp-async-7,c841ddb9f3cb9ffe,1a1ebeaa791283df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:32.448 WARN 2458 --- [ocp-async-4,312cc894a0396460,661cf549d382c96f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:32.551 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,29fd956d7cedf86e,f063744e0a5d90c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=29fd956d7cedf86e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 10:59:32.558 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,29fd956d7cedf86e,f063744e0a5d90c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=29fd956d7cedf86e, duration=15 ms] 2025-10-27 10:59:32.707 INFO 2458 --- [prometheus-minute-schedule-1,0ed9f0b51af8837d,3b380eaea596c6d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:32.714 INFO 2458 --- [prometheus-minute-schedule-1,0ed9f0b51af8837d,3b380eaea596c6d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:32.833 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:32.834 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:32.964 INFO 2458 --- [ocp-schedules-26,19a5afc0eccce078,72373eba64d950ed] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:32.967 INFO 2458 --- [ocp-schedules-26,19a5afc0eccce078,72373eba64d950ed] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:33.203 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,701ca1093a58d7ef,bd190d8e89f45a19] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=701ca1093a58d7ef, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 10:59:33.203 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,701ca1093a58d7ef,bd190d8e89f45a19] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=701ca1093a58d7ef, duration=7 ms] 2025-10-27 10:59:33.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,cf0a8736b9af32d7,efeb0e3c35aee271] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=cf0a8736b9af32d7, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 10:59:33.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,cf0a8736b9af32d7,efeb0e3c35aee271] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=cf0a8736b9af32d7, duration=9 ms] 2025-10-27 10:59:33.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,57ace374b6419050,ed9fde171ba94047] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=57ace374b6419050, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 10:59:33.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,57ace374b6419050,ed9fde171ba94047] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=57ace374b6419050, duration=9 ms] 2025-10-27 10:59:33.224 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 10:59:33.224 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 10:59:33.224 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 10:59:33.361 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 10:59:33.361 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 10:59:33.423 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:33.423 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:33.464 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:33.464 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:33.465 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:33.466 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 10:59:33.478 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:33.478 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:33.478 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:33.479 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 10:59:33.537 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:33.537 INFO 2458 --- [ocp-schedules-13,fec5008d07d51eb0,6d7fbaba6516abf1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 10:59:33.665 INFO 2458 --- [ocp-schedules-1,6ceb99b21d2220e5,666e66acb4f16ae7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 10:59:33.665 INFO 2458 --- [ocp-schedules-1,6ceb99b21d2220e5,666e66acb4f16ae7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 10:59:34.448 WARN 2458 --- [ocp-async-5,cc1a1b229c98a47e,9e10f911224fb8be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:34.448 INFO 2458 --- [ocp-async-2,30fbc405da712cd1,149c15eb52d91b79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:34.815 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,b534fdcc466aae54,c2d1184b1a727234] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clustersView?page=1&size=10, client=10.40.198.13, traceId=b534fdcc466aae54, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClustersView(String,Pageable), args=Page request [number: 0, size 10, sort: name: ASC],] 2025-10-27 10:59:34.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,b534fdcc466aae54,c2d1184b1a727234] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clustersView client=10.40.198.13, traceId=b534fdcc466aae54, duration=24 ms] 2025-10-27 10:59:34.835 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:34.835 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:59:35.470 INFO 2458 --- [ocp-schedules-11,509c894faf3568c2,14b1da86ad2dd936] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:35.651 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,e44dead743475f94,e5d3e725fff53106] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=e44dead743475f94, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 10:59:35.660 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,e44dead743475f94,e5d3e725fff53106] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=e44dead743475f94, duration=17 ms] 2025-10-27 10:59:35.707 INFO 2458 --- [prometheus-minute-schedule-1,b0bd6250b2a937c4,e31f871e27d93115] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:35.714 INFO 2458 --- [prometheus-minute-schedule-1,b0bd6250b2a937c4,e31f871e27d93115] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:35.834 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:35.835 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:36.122 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:36.122 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 10:59:36.123 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 10:59:36.123 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 10:59:36.123 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 10:59:36.123 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 10:59:36.135 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:36.136 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:36.136 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:36.136 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:36.136 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:36.140 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:36.140 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:36.140 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:36.140 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:36.140 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:36.201 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 10:59:36.206 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 10:59:36.210 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 10:59:36.212 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 10:59:36.222 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 10:59:36.225 INFO 2458 --- [ocp-schedules-10,9224e7c8dc25b2b3,e5e4ca8d3f8d7b5f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 10:59:36.448 INFO 2458 --- [ocp-async-10,0ff1558ad936925b,e0c76aeea1ab6de8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:36.448 WARN 2458 --- [ocp-async-8,bbe6f1a24b720071,2c6beddcc959eaad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:37.708 INFO 2458 --- [prometheus-cache-monitor1,efd013f143944276,b0b294121bf069aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20270,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5072,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7854} 2025-10-27 10:59:37.708 INFO 2458 --- [prometheus-cache-monitor1,efd013f143944276,b0b294121bf069aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 10:59:37.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c3ae0819d8c85804,5a4b7838b765a59d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=c3ae0819d8c85804, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 10:59:37.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c3ae0819d8c85804,5a4b7838b765a59d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=c3ae0819d8c85804, duration=10 ms] 2025-10-27 10:59:37.818 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,44a9b5a02b9f249d,306ea32e4905b259] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters?null, client=10.40.198.13, traceId=44a9b5a02b9f249d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClusters(Pageable), args=Page request [number: 0, size 5000, sort: name: ASC],] 2025-10-27 10:59:37.819 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,c653852b519b851f,1e6ec9c2f3880c03] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=c653852b519b851f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 10:59:37.821 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5469f95f6b43266c,7f44fa96775d5304] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=5469f95f6b43266c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 10:59:37.823 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=1f88288aa6f96518, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 10:59:37.827 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,c653852b519b851f,1e6ec9c2f3880c03] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=c653852b519b851f, duration=20 ms] 2025-10-27 10:59:37.841 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,44a9b5a02b9f249d,306ea32e4905b259] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters client=10.40.198.13, traceId=44a9b5a02b9f249d, duration=34 ms] 2025-10-27 10:59:37.844 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:37.845 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5469f95f6b43266c,7f44fa96775d5304] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=5469f95f6b43266c, duration=38 ms] 2025-10-27 10:59:37.848 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 10:59:37.860 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,973cb387a9630e93,d372915587ef1436] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=973cb387a9630e93, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 10:59:37.862 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,973cb387a9630e93,d372915587ef1436] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=973cb387a9630e93, duration=8 ms] 2025-10-27 10:59:37.874 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,85557b1e87d7f89a,6298b61ab5588056] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=85557b1e87d7f89a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 10:59:37.889 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,0b122777ca1273a1,6db9cfff873911c4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=0b122777ca1273a1, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 10:59:37.911 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,0b122777ca1273a1,6db9cfff873911c4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=0b122777ca1273a1, duration=28 ms] 2025-10-27 10:59:37.981 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:37.981 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:59:38.002 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 10:59:38.002 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-10,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 10:59:38.002 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 10:59:38.096 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,85557b1e87d7f89a,6298b61ab5588056] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=85557b1e87d7f89a, duration=229 ms] 2025-10-27 10:59:38.108 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1f88288aa6f96518,ec7c67f9a6ee6cb7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=1f88288aa6f96518, duration=301 ms] 2025-10-27 10:59:38.139 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c7f3e074c7123443,4856c60267c85561] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=c7f3e074c7123443, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 10:59:38.155 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c7f3e074c7123443,4856c60267c85561] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:38.155 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c7f3e074c7123443,4856c60267c85561] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 10:59:38.281 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c7f3e074c7123443,4856c60267c85561] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=c7f3e074c7123443, duration=148 ms] 2025-10-27 10:59:38.448 INFO 2458 --- [ocp-async-9,000ec93844567cad,579dd309991cf7ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:38.448 WARN 2458 --- [ocp-async-4,ca2c11e7e6e6e501,b26480a588e41faf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:38.502 INFO 2458 --- [ocp-async-7,579c42560ed8ec75,08cf2c154ba69122] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:38.533 INFO 2458 --- [ocp-schedules-26,6cf3ee7775c9674c,74626dd32ccc5129] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 10:59:38.538 INFO 2458 --- [ocp-schedules-26,6cf3ee7775c9674c,74626dd32ccc5129] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:38.539 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 10:59:38.539 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 10:59:38.539 INFO 2458 --- [ocp-schedules-26,6cf3ee7775c9674c,74626dd32ccc5129] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:59:38.707 INFO 2458 --- [prometheus-minute-schedule-1,361f9ccaf3c474f5,ed27d24b14d329af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:38.715 INFO 2458 --- [prometheus-minute-schedule-1,361f9ccaf3c474f5,ed27d24b14d329af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:38.835 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:38.836 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:39.795 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6db3e6872914c632,cad1a548c9f875b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=6db3e6872914c632, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 10:59:39.795 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6db3e6872914c632,cad1a548c9f875b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=6db3e6872914c632, duration=12 ms] 2025-10-27 10:59:39.800 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,49242ef1b7723ebd,bb80441618f4fd6f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=49242ef1b7723ebd, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 10:59:39.800 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,49242ef1b7723ebd,bb80441618f4fd6f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=49242ef1b7723ebd, duration=17 ms] 2025-10-27 10:59:39.800 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6c61e21548272b13,7cbc24ed8ca62623] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=6c61e21548272b13, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 10:59:39.800 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6c61e21548272b13,7cbc24ed8ca62623] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=6c61e21548272b13, duration=17 ms] 2025-10-27 10:59:39.821 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 10:59:39.821 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 10:59:39.821 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 10:59:40.011 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 10:59:40.011 INFO 2458 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 10:59:40.046 INFO 2458 --- [Check obproxy1,1db69f85c3c0c8c2,05097158c9cf64a1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:59:40.050 INFO 2458 --- [Check obproxy1,1db69f85c3c0c8c2,05097158c9cf64a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:59:40.050 INFO 2458 --- [Collect obproxy1,57232dac0be13186,1444b41b225a828e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 10:59:40.050 INFO 2458 --- [Collect obproxy1,57232dac0be13186,1444b41b225a828e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 10:59:40.058 INFO 2458 --- [ocp-schedules-7,d5df3d9b2b99bbb6,7da47ba082b93d89] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 10:59:40.059 INFO 2458 --- [ocp-schedules-7,d5df3d9b2b99bbb6,7da47ba082b93d89] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 10:59:40.086 INFO 2458 --- [Check obproxy1,1db69f85c3c0c8c2,05097158c9cf64a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 10:59:40.086 INFO 2458 --- [Collect obproxy1,57232dac0be13186,1444b41b225a828e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 10:59:40.089 INFO 2458 --- [Check obproxy1,1db69f85c3c0c8c2,05097158c9cf64a1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 10:59:40.089 INFO 2458 --- [Check obproxy1,1db69f85c3c0c8c2,05097158c9cf64a1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 10:59:40.090 INFO 2458 --- [Collect obproxy1,57232dac0be13186,1444b41b225a828e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 10:59:40.092 INFO 2458 --- [Collect obproxy1,57232dac0be13186,1444b41b225a828e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 10:59:40.448 INFO 2458 --- [ocp-async-5,a26d1d52096419e6,b2844ca4fae992e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:40.448 WARN 2458 --- [ocp-async-3,01c6376195ae4efa,30764366d39d7003] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:41.247 INFO 2458 --- [ocp-schedules-1,8b400baa737aa396,e364c00d299b832b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:41.326 WARN 2458 --- [scanner-112111810-1,8b400baa737aa396,81e57f07df463918] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:41.570 INFO 2458 --- [alarm-detect-7,8b400baa737aa396,e7ef75c00c63e033] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533955 ~ 1761533971 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533970, value=68.786)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533970, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533970=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 10:59:41.707 INFO 2458 --- [prometheus-minute-schedule-1,dc3fdf18f9c8b07f,3f38ad2d7abba328] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:41.715 INFO 2458 --- [prometheus-minute-schedule-1,dc3fdf18f9c8b07f,3f38ad2d7abba328] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:41.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:41.715168+08:00, lastProcessTime=2025-10-27T10:59:31.718793+08:00 2025-10-27 10:59:41.836 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:41.836 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:41.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:41.715168+08:00 2025-10-27 10:59:42.280 INFO 2458 --- [ocp-schedules-16,2378b0457f90413a,bcaf180fc9f14f70] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 10:59:42.280 INFO 2458 --- [ocp-schedules-16,2378b0457f90413a,bcaf180fc9f14f70] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 10:59:42.288 INFO 2458 --- [ocp-schedules-16,2378b0457f90413a,bcaf180fc9f14f70] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 10:59:42.288 INFO 2458 --- [ocp-schedules-16,2378b0457f90413a,bcaf180fc9f14f70] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 10:59:42.448 WARN 2458 --- [ocp-async-10,55ac6bb63cecf2ae,e3c782a5bff1b667] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:42.448 INFO 2458 --- [ocp-async-6,15013a8558c48f0d,45c790b334f545f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:42.784 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,331d316ceca0d36a,de2d272b1a707276] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=331d316ceca0d36a, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 10:59:42.788 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,331d316ceca0d36a,de2d272b1a707276] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=331d316ceca0d36a, duration=12 ms] 2025-10-27 10:59:42.975 INFO 2458 --- [ocp-schedules-9,967743f822407e78,0832f5673c4d3064] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:42.979 INFO 2458 --- [ocp-schedules-9,967743f822407e78,0832f5673c4d3064] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:43.899 INFO 2458 --- [ocp-schedules-24,79baf4b0d09cded8,cdee72c7978298eb] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 10:59:44.448 WARN 2458 --- [ocp-async-4,1249f5307729f505,af728fdee85c702a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:44.448 INFO 2458 --- [ocp-async-1,1bc43a8b9e24e6b9,fbebfbcfb8f90e6d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:44.707 INFO 2458 --- [prometheus-minute-schedule-1,203ab9f053fc4bdd,e15b357f73018a2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:44.715 INFO 2458 --- [prometheus-minute-schedule-1,203ab9f053fc4bdd,e15b357f73018a2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:44.836 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:44.837 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:45.102 INFO 2458 --- [prometheus-second-schedule-1,baeee399ff535988,2ec2fdd768ec44fe] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 10:59:45.165 INFO 2458 --- [ocp-schedules-30,2c87a010387a66bf,0c896af6a32bd1c8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 10:59:45.165 INFO 2458 --- [ocp-schedules-30,2c87a010387a66bf,0c896af6a32bd1c8] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 10:59:45.225 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 10:59:45.226 INFO 2458 --- [pool-875-thread-2,2c87a010387a66bf,5f8a695c226c1684] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 10:59:45.269 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:45.269 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:45.292 INFO 2458 --- [pool-875-thread-2,2c87a010387a66bf,5f8a695c226c1684] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:45.293 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:45.293 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:45.293 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:45.296 INFO 2458 --- [pool-875-thread-2,2c87a010387a66bf,5f8a695c226c1684] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761533684000000, 1761533985000000] 2025-10-27 10:59:45.296 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761533684000000, 1761533985000000] 2025-10-27 10:59:45.340 WARN 2458 --- [pool-875-thread-2,2c87a010387a66bf,5f8a695c226c1684] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 10:59:45.360 INFO 2458 --- [pool-875-thread-2,2c87a010387a66bf,5f8a695c226c1684] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 10:59:45.429 WARN 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 10:59:45.429 WARN 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 10:59:45.429 WARN 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 10:59:45.438 INFO 2458 --- [pool-875-thread-1,2c87a010387a66bf,1b67b1c23698dd49] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 10:59:45.439 INFO 2458 --- [ocp-schedules-30,2c87a010387a66bf,0c896af6a32bd1c8] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 10:59:45.477 INFO 2458 --- [ocp-schedules-3,16d566fce1f419e2,cfff0feda731aeb9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:46.432 INFO 2458 --- [ocp-schedules-13,2c405e90b20b85b2,53a5359bd59c85dd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 10:59:46.440 INFO 2458 --- [ocp-schedules-13,2c405e90b20b85b2,53a5359bd59c85dd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:46.440 INFO 2458 --- [ocp-schedules-13,2c405e90b20b85b2,53a5359bd59c85dd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 10:59:46.449 INFO 2458 --- [ocp-async-7,af2acab0bc6e1014,b14aafc616523232] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:46.449 WARN 2458 --- [ocp-async-2,681ef28428bc4b13,22b0835c99e5e5fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:47.088 INFO 2458 --- [ocp-schedules-7,201bd3e6d53ccd39,659bd86fac3cb8f9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T10:59:47.088+08:00, clusters=[1, 2] 2025-10-27 10:59:47.102 INFO 2458 --- [ocp-schedules-7,201bd3e6d53ccd39,659bd86fac3cb8f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 10:59:47.104 INFO 2458 --- [ocp-schedules-7,201bd3e6d53ccd39,659bd86fac3cb8f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 10:59:47.707 INFO 2458 --- [prometheus-minute-schedule-1,224b8bfea825884b,99597054bd6ebde6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:47.714 INFO 2458 --- [prometheus-minute-schedule-1,224b8bfea825884b,99597054bd6ebde6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:47.837 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:47.838 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:48.449 WARN 2458 --- [ocp-async-8,e1820fa71f396f59,5f53091d2534c662] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:48.449 INFO 2458 --- [ocp-async-5,97de9ba603cade37,cd77246b11c6ca51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:48.496 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3e460b53a97763a6,1c3955eb9cec3310] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=3e460b53a97763a6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 10:59:48.496 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3e460b53a97763a6,1c3955eb9cec3310] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=3e460b53a97763a6, duration=1 ms] 2025-10-27 10:59:48.502 INFO 2458 --- [ocp-async-6,11eca85ee67b271b,9765a51a1f61682e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:50.449 WARN 2458 --- [ocp-async-1,8eb0280513bacc94,c9f167c52da83a29] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:50.449 INFO 2458 --- [ocp-async-9,56ddeee2dfea16f3,894113800ba08318] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:50.707 INFO 2458 --- [prometheus-minute-schedule-1,c8b01e3ceff03cba,678248bde49868f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:50.719 INFO 2458 --- [prometheus-minute-schedule-1,c8b01e3ceff03cba,678248bde49868f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:50.838 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:50.839 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:51.410 INFO 2458 --- [ocp-schedules-27,d6e4bca3336c2d5b,86c7b24e93db45f1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:51.410 INFO 2458 --- [ocp-schedules-12,a25e02924fc74b93,c42d9c4e3bf053d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:51.410 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:59:51.411 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:59:51.411 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 10:59:51.411 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 10:59:51.411 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:59:51.411 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:59:51.413 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:59:51.414 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 10:59:51.414 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 10:59:51.414 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 10:59:51.426 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:51.426 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:51.426 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:51.427 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:51.428 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:51.428 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:51.429 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:51.429 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:51.430 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:51.430 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:51.430 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:51.431 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:51.431 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:51.432 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:51.432 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:51.433 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:51.433 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:51.433 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:51.434 INFO 2458 --- [ocp-schedules-13,a8a1118dfef7cc3b,89a636d7f513119d] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 10:59:51.434 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:51.435 INFO 2458 --- [host-check-clockdiff1,a8a1118dfef7cc3b,05218f78ddd61873] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 10:59:51.437 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:51.438 INFO 2458 --- [host-check-clockdiff3,a8a1118dfef7cc3b,ee142f23a82e8cba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 10:59:51.438 INFO 2458 --- [host-check-clockdiff4,a8a1118dfef7cc3b,c9076df8788d31f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 10:59:51.438 INFO 2458 --- [host-check-clockdiff2,a8a1118dfef7cc3b,63a671a01e16d905] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 10:59:51.441 INFO 2458 --- [host-check-clockdiff5,a8a1118dfef7cc3b,3bf1f17b93aa7f59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 10:59:51.442 INFO 2458 --- [ocp-schedules-16,a6cc0a8759506dbc,d53fe4861c0597c6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 10:59:51.458 INFO 2458 --- [host-check-clockdiff3,a8a1118dfef7cc3b,ee142f23a82e8cba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533991 0 0 , error= 2025-10-27 10:59:51.458 INFO 2458 --- [host-check-clockdiff1,a8a1118dfef7cc3b,05218f78ddd61873] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533991 0 0 , error= 2025-10-27 10:59:51.458 INFO 2458 --- [host-check-clockdiff4,a8a1118dfef7cc3b,c9076df8788d31f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533991 -1 -1 , error= 2025-10-27 10:59:51.467 INFO 2458 --- [host-check-clockdiff5,a8a1118dfef7cc3b,3bf1f17b93aa7f59] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533991 0 0 , error= 2025-10-27 10:59:51.501 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 10:59:51.501 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 10:59:51.502 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T10:59:51.502112274+08:00 2025-10-27 10:59:51.502 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 10:59:51.502 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 10:59:51.503 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T10:59:51.502+08:00 2025-10-27 10:59:51.503 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 10:59:51.503 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T10:59:51.503812565+08:00 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T10:59:51.505+08:00 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T10:59:51.505692215+08:00 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 10:59:51.505 INFO 2458 --- [host-check-agent-status1,a25e02924fc74b93,7467d88c6ffeee45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status3,a25e02924fc74b93,973ac2eaf8d29e21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-agent-status2,a25e02924fc74b93,2e5dc8d315dc3e2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 10:59:51.506 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T10:59:51.505+08:00 2025-10-27 10:59:51.507 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 10:59:51.508 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 10:59:51.509 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T10:59:51.509158529+08:00 2025-10-27 10:59:51.510 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T10:59:51.509+08:00 2025-10-27 10:59:51.510 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 10:59:51.510 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 10:59:51.510 INFO 2458 --- [host-check-agent-status5,a25e02924fc74b93,58f0c8b3e998d4d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 10:59:51.511 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 10:59:51.512 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T10:59:51.511520783+08:00 2025-10-27 10:59:51.513 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T10:59:51.512+08:00 2025-10-27 10:59:51.515 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 10:59:51.517 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T10:59:51.502+08:00 and status AVAILABLE 2025-10-27 10:59:51.517 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T10:59:51.509+08:00 and status AVAILABLE 2025-10-27 10:59:51.517 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 10:59:51.517 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 10:59:51.517 INFO 2458 --- [host-check-agent-status4,a25e02924fc74b93,95ff4e91fe42ae38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 10:59:51.520 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T10:59:51.512+08:00 and status AVAILABLE 2025-10-27 10:59:51.520 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T10:59:51.505+08:00 and status AVAILABLE 2025-10-27 10:59:51.521 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T10:59:51.505+08:00 and status AVAILABLE 2025-10-27 10:59:51.522 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 10:59:51.523 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 10:59:51.525 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 10:59:51.526 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 10:59:51.528 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 10:59:51.543 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 10:59:51.545 INFO 2458 --- [ocp-schedules-5,558e2af5eb92201f,50c17eaf3d6cde8c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 10:59:51.546 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 10:59:51.547 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:51.548 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 10:59:51.548 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 10:59:51.549 INFO 2458 --- [ocp-schedules-18,1e42d90325763314,9061cf3851feb3f1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 10:59:51.550 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:51.551 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:51.551 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 10:59:51.551 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 10:59:51.551 INFO 2458 --- [ocp-schedules-5,558e2af5eb92201f,50c17eaf3d6cde8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:51.552 INFO 2458 --- [ocp-schedules-18,1e42d90325763314,9061cf3851feb3f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:51.553 INFO 2458 --- [ocp-schedules-18,1e42d90325763314,9061cf3851feb3f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 10:59:51.553 INFO 2458 --- [ocp-schedules-15,34a7467dabed77e4,63b261ebb5715ad5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 10:59:51.555 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:51.561 INFO 2458 --- [ocp-schedules-15,34a7467dabed77e4,63b261ebb5715ad5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:51.566 INFO 2458 --- [ocp-schedules-6,0616bffe4fa6babf,dc2c028aa167adb0] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 10:59:51.568 INFO 2458 --- [ocp-schedules-29,7f5fd4eb771c88cc,8c50e207d5ddafac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 10:59:51.569 INFO 2458 --- [ocp-schedules-25,cbcf9ae0d98f42ee,ed55161994868b2f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 10:59:51.576 INFO 2458 --- [ocp-schedules-23,e6c88bf23c798500,22bb6fa9987df93d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 10:59:51.580 INFO 2458 --- [ocp-schedules-29,7f5fd4eb771c88cc,8c50e207d5ddafac] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 10:59:51.624 INFO 2458 --- [ocp-schedules-25,cbcf9ae0d98f42ee,ed55161994868b2f] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 10:59:51.656 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.657 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:51.658 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 10:59:51.681 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:51.681 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:51.688 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.688 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.688 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.688 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.689 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.690 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:51.693 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:51.702 WARN 2458 --- [scanner-112112054-1,e6c88bf23c798500,b950cc13c89ce829] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 10:59:51.703 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 10:59:51.703 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 10:59:51.710 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:51.711 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 10:59:51.719 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 10:59:51.721 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.722 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:51.725 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T10:59:51.723727+08:00, lastProcessTime=2025-10-27T10:59:41.715168+08:00 2025-10-27 10:59:51.729 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 10:59:51.737 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 10:59:51.752 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 10:59:51.761 INFO 2458 --- [ocp-schedules-5,558e2af5eb92201f,50c17eaf3d6cde8c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:59:51.779 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 10:59:51.782 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 10:59:51.796 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 10:59:51.800 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 10:59:51.818 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.818 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:51.822 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.822 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.823 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.823 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:59:51.836 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:51.838 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 10:59:51.843 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:51.851 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:51.851 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:51.859 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 10:59:51.869 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 10:59:51.872 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 10:59:51.876 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 10:59:51.888 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 10:59:51.906 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 10:59:51.909 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 10:59:51.909 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 10:59:51.910 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 10:59:51.920 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:51.920 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.921 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:51.926 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.927 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 10:59:51.930 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:59:51.931 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:51.941 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:51.941 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.941 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 10:59:51.944 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:51.946 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:51.951 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:51.961 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 10:59:51.964 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:51.965 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 10:59:51.965 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:51.965 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:51.967 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 10:59:51.968 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 10:59:51.994 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 10:59:51.995 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 10:59:52.000 INFO 2458 --- [host-check-host-status3,d6e4bca3336c2d5b,74e166cade021fa8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 10:59:52.004 INFO 2458 --- [host-check-host-status2,d6e4bca3336c2d5b,19259f97de5a0465] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 10:59:52.008 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 10:59:52.008 INFO 2458 --- [host-check-host-status5,d6e4bca3336c2d5b,dc31a6bdf93d43f1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 10:59:52.013 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.034 INFO 2458 --- [host-check-host-status1,d6e4bca3336c2d5b,396811fa110cfc83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 10:59:52.042 INFO 2458 --- [host-check-host-status4,d6e4bca3336c2d5b,8692badfa6528bcc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 10:59:52.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:52.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.043 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.043 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 10:59:52.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.044 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.047 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 10:59:52.048 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:52.049 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:52.049 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:52.050 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 10:59:52.060 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 10:59:52.071 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 10:59:52.073 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 10:59:52.088 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T10:59:51.723727+08:00 2025-10-27 10:59:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:52.136 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:59:52.140 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:52.180 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 10:59:52.180 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.181 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:59:52.187 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.188 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:59:52.204 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:59:52.217 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.217 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 10:59:52.227 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:52.232 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.233 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:59:52.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 10:59:52.256 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:59:52.256 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 10:59:52.290 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.334 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 10:59:52.339 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 10:59:52.351 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 10:59:52.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 10:59:52.372 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 10:59:52.383 INFO 2458 --- [host-check-clockdiff2,a8a1118dfef7cc3b,63a671a01e16d905] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761533992 -1 -1 , error= 2025-10-27 10:59:52.425 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 10:59:52.449 WARN 2458 --- [ocp-async-3,6f74817c3bb4467b,acf833ceff236dec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:52.449 INFO 2458 --- [ocp-async-8,f6570ac0879a2b77,68e9c26205085433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:52.480 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 10:59:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.481 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.481 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 10:59:52.483 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.483 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.483 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.484 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 10:59:52.520 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.567 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.568 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 10:59:52.611 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:59:52.611 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533402507000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 10:59:52.611 INFO 2458 --- [ocp-schedules-15,34a7467dabed77e4,63b261ebb5715ad5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 10:59:52.759 INFO 2458 --- [ocp-schedules-25,8bef0105785f6dd7,808f3104136287be] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 10:59:52.765 INFO 2458 --- [ocp-schedules-25,8bef0105785f6dd7,808f3104136287be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:52.851 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:59:52.851 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.854 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:59:52.866 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.866 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:52.897 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.897 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:52.929 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.930 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:52.932 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.933 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:52.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:59:52.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.936 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.939 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 10:59:52.958 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.958 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.958 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.959 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:52.960 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.961 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:52.986 INFO 2458 --- [ocp-schedules-17,af5024c0d6747294,3df3816fa3b3b50b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 10:59:52.990 INFO 2458 --- [ocp-schedules-17,af5024c0d6747294,3df3816fa3b3b50b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 10:59:52.992 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:52.992 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:52.992 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.992 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:52.992 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:52.993 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:53.014 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.014 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.014 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.015 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 10:59:53.017 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.017 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.018 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.018 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:53.028 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.029 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.030 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.030 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.031 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.032 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.033 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.035 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.036 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.037 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.038 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 10:59:53.045 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:59:53.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.048 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.048 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:53.074 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.074 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.075 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.075 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 10:59:53.077 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.077 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.078 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.078 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:59:53.096 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:53.097 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.098 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 10:59:53.108 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:59:53.111 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 10:59:53.111 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 10:59:53.139 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.139 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.139 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.140 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 10:59:53.144 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.144 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.144 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.145 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:59:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 10:59:53.173 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.173 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.174 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.174 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:59:53.203 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.203 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.203 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.204 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 10:59:53.218 INFO 2458 --- [ocp-schedules-5,87571569eff0a36a,eb63783fe697b8f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20270,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5074,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7854} 2025-10-27 10:59:53.218 INFO 2458 --- [ocp-schedules-5,87571569eff0a36a,eb63783fe697b8f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 10:59:53.250 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.250 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.253 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.253 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.253 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.256 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.256 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.257 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.257 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.259 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.259 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.261 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.261 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.263 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.264 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.264 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.264 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 10:59:53.265 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.265 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.265 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 10:59:53.267 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.267 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.268 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 10:59:53.268 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 10:59:53.271 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 10:59:53.359 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 10:59:53.360 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 10:59:53.388 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 10:59:53.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 10:59:53.392 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 10:59:53.392 INFO 2458 --- [ocp-schedules-25,8bef0105785f6dd7,808f3104136287be] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 10:59:53.707 INFO 2458 --- [prometheus-minute-schedule-1,d0c44eb9e9953c4f,bc76de7c8a5093d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:53.714 INFO 2458 --- [prometheus-minute-schedule-1,d0c44eb9e9953c4f,bc76de7c8a5093d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:53.839 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:53.840 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:54.053 INFO 2458 --- [ocp-schedules-24,9df5e00d46ace8f9,6ffb81ec6dccdb0b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 10:59:54.449 WARN 2458 --- [ocp-async-8,8e709a50fc0ab8a5,313bf8e632e292b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:54.449 INFO 2458 --- [ocp-async-6,569c6588f9b94278,d1cac54decac8a51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:55.485 INFO 2458 --- [ocp-schedules-29,70c855be3b5ddae6,17094743ef82c9bc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 10:59:56.449 WARN 2458 --- [ocp-async-6,ae7d189116c0de0a,6dfafb682f0dad8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:56.449 INFO 2458 --- [ocp-async-4,73197c77d25a5a7d,447052f1d8cffbe7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:56.707 INFO 2458 --- [prometheus-minute-schedule-1,d3426cf31410f648,d6b347588b34fb2b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:56.714 INFO 2458 --- [prometheus-minute-schedule-1,d3426cf31410f648,d6b347588b34fb2b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:56.840 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:56.841 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 10:59:57.475 INFO 2458 --- [ocp-schedules-9,1272917f43a822df,827ba1a45b77e139] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 10:59:57.483 INFO 2458 --- [ocp-schedules-9,1272917f43a822df,827ba1a45b77e139] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 10:59:57.530 INFO 2458 --- [ocp-schedules-9,1272917f43a822df,827ba1a45b77e139] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 10:59:58.449 WARN 2458 --- [ocp-async-3,af50843ea1c50629,943150057d8d6364] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 10:59:58.449 INFO 2458 --- [ocp-async-7,6fde82dbe7b89266,ea7d8f9c18130f42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 10:59:58.502 INFO 2458 --- [ocp-async-8,8d1385c7fa6bd2ed,71dd5b9192dd3fc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 10:59:59.044 INFO 2458 --- [ocp-schedules-29,bf17d6747961bcd9,ede900e6c6ebbf09] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 10:59:59.707 INFO 2458 --- [prometheus-minute-schedule-1,3a2c24d178c52b78,47a17b6b0d98af02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534000, interval=60 2025-10-27 10:59:59.714 INFO 2458 --- [prometheus-minute-schedule-1,3a2c24d178c52b78,47a17b6b0d98af02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 10:59:59.841 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 10:59:59.841 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:00.107 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 10:55:01.0, nextRun=2025-10-27 11:00:00.0, currentDate=Mon Oct 27 11:00:00 CST 2025 2025-10-27 11:00:00.107 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 10:55:01.0, nextRunTime=2025-10-27 11:00:00.0} 2025-10-27 11:00:00.145 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:00:00.145 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:00:00.145+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:00:00.175 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 10:55:01.0, nextRun=2025-10-27 11:00:00.0, currentDate=Mon Oct 27 11:00:00 CST 2025 2025-10-27 11:00:00.175 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 10:55:01.0, nextRunTime=2025-10-27 11:00:00.0} 2025-10-27 11:00:00.270 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:00:00.309 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:00:00.317 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:00:00.317 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,25a8cfe58a76df59] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:00:00.317+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:00:00.356 INFO 2458 --- [subtask-coordinator1,8964d58b78e8b300,3de179b1407386a2] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9141, operation:EXECUTE, state: RUNNING 2025-10-27 11:00:00.399 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:00:00.400 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9141, context=Context{parallelIdx=-1, stringMap={task_instance_id=9090, task_operation=execute, latest_execution_start_time=2025-10-27T11:00:00.342+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9141}, listMap={}}, executor=10.40.65.146 2025-10-27 11:00:00.450 WARN 2458 --- [ocp-async-8,086df262de8f36db,127be73f8f7313c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:00.450 INFO 2458 --- [ocp-async-1,84e2714d7ddf96c0,2890988384ff3bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:00.486 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:00.618 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:00.704 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:00.783 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:00.957 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:01.088 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:01.348 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:01.377 INFO 2458 --- [subtask-coordinator1,8964d58b78e8b300,67fddb8badb3a211] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9142, operation:EXECUTE, state: RUNNING 2025-10-27 11:00:01.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:00:01.413 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9142, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9091, task_operation=execute, latest_execution_start_time=2025-10-27T11:00:01.374+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9142}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:00:01.413 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:00:01.419 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:00:01.435 INFO 2458 --- [ocp-schedules-23,dad3cda6904832ba,db35d4690abfbac5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:00:01.441 INFO 2458 --- [ocp-schedules-23,dad3cda6904832ba,db35d4690abfbac5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:01.457 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:00:01.458 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:00:01.464 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.494 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.500 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:01.501 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:00:01.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:01.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:01.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:01.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:01.504 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:01.504 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:00:01.512 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.524 INFO 2458 --- [ocp-schedules-23,dad3cda6904832ba,db35d4690abfbac5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:00:01.537 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.590 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.614 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:00:01.619 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:00:01.638 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:01.717125+08:00, lastProcessTime=2025-10-27T10:59:51.723727+08:00 2025-10-27 11:00:01.720 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.759 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:00:01.764 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:00:01.771 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:00:01.783 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:00:01.783 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:00:01.801 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,71ef29abbe170e93] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9141, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:00:01.893 INFO 2458 --- [ocp-schedules-11,341e3c74cce70826,261bb0e0fba0df92] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:01.937 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:00:01.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:00:01.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:00:01.997 WARN 2458 --- [scanner-112112425-1,341e3c74cce70826,f6fadef7790215c2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:02.038 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:00:02.044 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:00:02.054 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,8df32739160c0244] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9142, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:00:02.093 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:01.717125+08:00 2025-10-27 11:00:02.283 INFO 2458 --- [alarm-detect-7,341e3c74cce70826,61446f5049198980] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533970 ~ 1761533992 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533985, value=56.778)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761533985, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761533985=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:00:02.285 INFO 2458 --- [alarm-detect-7,341e3c74cce70826,61446f5049198980] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:00:02.380 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9090, status:SUCCESS 2025-10-27 11:00:02.396 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9090, status:SUCCESS 2025-10-27 11:00:02.405 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9091, status:SUCCESS 2025-10-27 11:00:02.406 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9091, status:SUCCESS 2025-10-27 11:00:02.450 WARN 2458 --- [ocp-async-6,0a63f53a0fbac9b7,45819be8ddbbf161] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:02.450 INFO 2458 --- [ocp-async-2,aead4a8e471573a8,d0de176b91c87e6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:02.507 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1816509362a82ac7,8c42820bb81e7c89] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&version=2&ObClusterId=1760510187, client=10.40.65.146, traceId=1816509362a82ac7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase,1760510187,2,{"ObRegion":"myoceanbase","ObCluster":"myoceanbase","ObRegionId":1760510187,"ObClusterId":1760510187,"Type":"PRIMARY","timestamp":1761534002505733,"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:00:02.523 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1816509362a82ac7,8c42820bb81e7c89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase, obClusterId=1760510187, rootServerJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, groupName=myoceanbase, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:00:02.523 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1816509362a82ac7,8c42820bb81e7c89] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:00:02.523 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1816509362a82ac7,8c42820bb81e7c89] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase, clusterId=1760510187, affectRows=1 2025-10-27 11:00:02.524 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,1816509362a82ac7,8c42820bb81e7c89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.65.146, traceId=1816509362a82ac7, duration=18 ms] 2025-10-27 11:00:02.707 INFO 2458 --- [prometheus-minute-schedule-1,a1decfb5fd38c7b6,6163ba3ef37b2201] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:02.717 INFO 2458 --- [prometheus-minute-schedule-1,a1decfb5fd38c7b6,6163ba3ef37b2201] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:02.842 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:02.842 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:02.999 INFO 2458 --- [ocp-schedules-29,27346c90166fa902,939834a4352f62fc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:03.003 INFO 2458 --- [ocp-schedules-29,27346c90166fa902,939834a4352f62fc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:04.450 WARN 2458 --- [ocp-async-7,a0a9a32fa7717cfa,7d6f66f2fa2aea34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:04.450 INFO 2458 --- [ocp-async-10,5a0755d351fe7cd7,6b33d07b12f933cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:04.462 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,67cf535dc8b6d19a] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 10:50:05.0, nextRun=2025-10-27 11:00:04.0, currentDate=Mon Oct 27 11:00:04 CST 2025 2025-10-27 11:00:04.462 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,67cf535dc8b6d19a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-10-27 10:50:05.0, nextRunTime=2025-10-27 11:00:04.0} 2025-10-27 11:00:04.471 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,67cf535dc8b6d19a] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-10-27 11:00:04.471 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,67cf535dc8b6d19a] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:00:04.471+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:00:05.405 INFO 2458 --- [subtask-coordinator1,8964d58b78e8b300,477c713eab3b910f] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9143, operation:EXECUTE, state: RUNNING 2025-10-27 11:00:05.437 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:00:05.438 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9143, context=Context{parallelIdx=-1, stringMap={task_instance_id=9092, task_operation=execute, latest_execution_start_time=2025-10-27T11:00:05.403+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=9143}, listMap={}}, executor=10.40.65.146 2025-10-27 11:00:05.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:00:05.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:00:05.447 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.448 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:00:05.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.451 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.452 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:00:05.459 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7de95c1f7ca936ad,eec3de20963ef841] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=7de95c1f7ca936ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:00:05.459 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7de95c1f7ca936ad,eec3de20963ef841] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=7de95c1f7ca936ad, duration=1 ms] 2025-10-27 11:00:05.464 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.464 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.465 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.465 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:00:05.471 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:00:05.471 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:00:05.473 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.473 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:00:05.475 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.476 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.476 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.477 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:00:05.495 INFO 2458 --- [ocp-schedules-13,36d4ea463b5e6e89,3a3413bb09e0c709] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:05.496 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.497 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.498 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.498 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:00:05.514 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.514 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.515 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.516 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:00:05.530 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.531 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.532 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.533 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:00:05.547 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.548 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.548 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.549 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:00:05.562 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.563 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.563 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.564 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:00:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.580 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.581 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:00:05.594 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.595 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.596 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.596 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:00:05.603 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.604 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.605 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.605 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:00:05.612 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.613 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.614 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.615 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:00:05.620 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.621 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.622 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.622 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:00:05.628 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.628 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.629 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.630 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:00:05.636 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.637 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.637 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.638 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:00:05.645 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.646 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.647 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.647 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:00:05.654 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.654 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.655 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.656 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:00:05.662 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.663 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.664 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.664 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:00:05.671 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.671 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.672 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.673 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:00:05.679 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.680 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.680 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.681 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:00:05.690 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.690 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.691 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.692 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:00:05.701 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.702 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.703 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.703 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:00:05.707 INFO 2458 --- [prometheus-minute-schedule-1,eca862fe4eec0ae9,088dedd81eefc19b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:05.714 INFO 2458 --- [prometheus-minute-schedule-1,eca862fe4eec0ae9,088dedd81eefc19b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:05.717 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.717 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.718 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.719 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:00:05.727 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.727 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.728 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.729 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:00:05.736 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.736 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.737 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.738 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:00:05.744 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.744 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.745 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.746 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:00:05.751 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.752 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.752 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.753 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:00:05.759 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.760 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.760 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.761 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:00:05.767 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.767 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.768 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.768 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:00:05.775 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.776 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.776 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.777 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:00:05.782 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.783 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.784 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.784 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:00:05.790 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.790 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.791 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.792 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:00:05.797 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.798 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.799 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.799 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:00:05.805 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.805 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.806 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.807 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:00:05.812 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.813 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.814 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.814 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:00:05.819 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.820 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.820 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.821 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:00:05.827 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.827 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.828 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.829 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:00:05.834 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.834 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.835 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.836 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:00:05.841 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.842 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.842 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.842 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:05.843 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:00:05.843 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:05.848 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.849 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.850 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.850 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:00:05.856 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.857 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.858 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.858 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:00:05.864 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.865 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.865 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.866 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:00:05.872 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.872 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.873 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.874 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:00:05.879 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.879 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.880 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.881 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:00:05.888 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.889 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.890 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.890 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:00:05.897 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.898 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.898 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.899 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:00:05.904 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.905 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.905 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.906 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:00:05.911 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.912 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.912 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.913 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:00:05.919 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.919 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.920 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.921 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:00:05.925 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.926 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.926 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.927 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:00:05.932 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.932 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.933 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.934 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:00:05.938 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.940 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:00:05.945 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.945 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.946 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.946 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:00:05.951 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.952 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.953 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.953 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:00:05.959 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.960 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.960 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.961 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:00:05.966 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.967 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.967 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.968 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:00:05.973 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.974 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.975 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.975 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:00:05.979 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.980 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.980 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.981 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:00:05.986 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.986 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:05.987 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.988 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:00:05.992 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:00:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:00:06.001 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:06.001 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:00:06.003 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:00:06.004 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,73f96f2dee5a0807] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9143, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:00:06.055 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:00:06.095 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:00:06.119 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:00:06.119 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:00:06.119 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:00:06.119 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:00:06.122 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:00:06.123 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:00:06.122 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:00:06.123 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:00:06.123 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:00:06.123 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:00:06.123 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:06.124 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:00:06.126 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:06.126 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:00:06.443 INFO 2458 --- [ocp-schedules-19,f6fffdb02a60b22d,805cd36174f36f97] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:00:06.448 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9092, status:SUCCESS 2025-10-27 11:00:06.449 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9092, status:SUCCESS 2025-10-27 11:00:06.450 WARN 2458 --- [ocp-async-5,377ee77e637e2294,addc4bf8a52f37f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:06.450 INFO 2458 --- [ocp-async-9,63d0846309971932,2dfba7402c2b2a35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:06.451 INFO 2458 --- [ocp-schedules-19,f6fffdb02a60b22d,805cd36174f36f97] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:06.451 INFO 2458 --- [ocp-schedules-19,f6fffdb02a60b22d,805cd36174f36f97] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:06.547 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T10:57:30, start=2025-10-27T10:55:00, end=2025-10-27T10:57:30, updates=258, granule=120000ms, delay=150000ms 2025-10-27 11:00:06.554 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T10:57:30, target=2025-10-27T10:57:30, granule=120000ms, delay=0ms) 2025-10-27 11:00:06.666 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - source=2025-10-27T10:57:30, start=2025-10-27T10:42:30, end=2025-10-27T10:52:30, updates=299, granule=600000ms, delay=900000ms 2025-10-27 11:00:06.675 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=1) - not enough cached interval (source=2025-10-27T10:57:30, target=2025-10-27T10:52:30, granule=600000ms, delay=300000ms) 2025-10-27 11:00:06.675 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=672ms, period=120000ms 2025-10-27 11:00:08.450 WARN 2458 --- [ocp-async-1,001b187acfd6d1b3,530bdf319c9a29bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:08.450 INFO 2458 --- [ocp-async-3,edffada1e52cc4bb,7dad5c91826a580c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:08.503 INFO 2458 --- [ocp-async-7,17e8f09dcb101575,5a2aa0f615cf9000] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:08.707 INFO 2458 --- [prometheus-minute-schedule-1,5df955a307646579,807d9a8e5e33d0c5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:08.715 INFO 2458 --- [prometheus-minute-schedule-1,5df955a307646579,807d9a8e5e33d0c5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:08.843 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:08.844 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:10.010 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:00:10.010 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:00:10.057 INFO 2458 --- [Check obproxy1,9d57ed1c4c84945b,28e89bf5499d7008] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:00:10.057 INFO 2458 --- [Check obproxy1,9d57ed1c4c84945b,28e89bf5499d7008] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:00:10.059 INFO 2458 --- [Collect obproxy1,48fe071c89ebebd4,a661851932d3a477] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:00:10.059 INFO 2458 --- [Collect obproxy1,48fe071c89ebebd4,a661851932d3a477] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:00:10.086 INFO 2458 --- [Collect obproxy1,48fe071c89ebebd4,a661851932d3a477] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:00:10.110 INFO 2458 --- [Collect obproxy1,48fe071c89ebebd4,a661851932d3a477] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:00:10.111 INFO 2458 --- [Check obproxy1,9d57ed1c4c84945b,28e89bf5499d7008] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:00:10.114 INFO 2458 --- [Collect obproxy1,48fe071c89ebebd4,a661851932d3a477] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:00:10.114 INFO 2458 --- [Check obproxy1,9d57ed1c4c84945b,28e89bf5499d7008] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:00:10.114 INFO 2458 --- [Check obproxy1,9d57ed1c4c84945b,28e89bf5499d7008] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:00:10.186 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:00:10.450 INFO 2458 --- [ocp-async-4,a4c42f2861e6538e,05f5732dd6ee75b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:10.450 WARN 2458 --- [ocp-async-10,a54ca82420cd1796,82dd140a6c46d995] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:11.099 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09} 2025-10-27 11:00:11.115 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView= 2025-10-27 11:00:11.115 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.117 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.117 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:00:11.117 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.120 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.122 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.122 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:00:11.122 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.124 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.135 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 4 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.172 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:00:11.172 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.289 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447611289000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:00:11.289Z, endTime=2025-10-27T02:58:11.090Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:00:11.312 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:00:11.312 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:00:11.350 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.350 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.406 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:00:11.406 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:00:11.431 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.431 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.436 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:00:11.436 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:00:11.289, end=2025-10-27T10:58:11.09}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:00:11.436 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.438 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.438 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:00:11.438 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.440 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.443 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.443 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:00:11.443 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.554 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447611554000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:00:11.554Z, endTime=2025-10-27T02:58:11.090Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:00:11.563 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:00:11.563 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:00:11.586 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.586 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.592 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:00:11.592 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:00:11.614 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.614 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.618 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:00:11.618 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:00:11.554, end=2025-10-27T10:58:11.09}, clusterId=2, tenantId=6, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:00:11.707 INFO 2458 --- [prometheus-minute-schedule-1,1079f087d4016983,70a19be3893bf86b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:11.714022+08:00, lastProcessTime=2025-10-27T11:00:01.717125+08:00 2025-10-27 11:00:11.715 INFO 2458 --- [prometheus-minute-schedule-1,1079f087d4016983,70a19be3893bf86b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:11.806 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447611805000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:00:11.805Z, endTime=2025-10-27T02:58:11.090Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:00:11.844 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:11.845 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:11.850 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 259 2025-10-27 11:00:11.850 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:00:11.875 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.875 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.893 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:00:11.893 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:00:11.918 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:00:11.918 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761533891090000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:00:11.923 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:00:11.952 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:00:11.805, end=2025-10-27T10:58:11.09}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:00:11.952 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:00:11.954 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:10.811, end=2025-10-27T10:58:11.09},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:00:11.954 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:00:11.954 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:00:11.956 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:00:12.304 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:11.714022+08:00 2025-10-27 11:00:12.327 INFO 2458 --- [ocp-schedules-15,62ec020a7b83acfa,78a90ca402388fe2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:12.353 WARN 2458 --- [scanner-112114186-1,62ec020a7b83acfa,d62065d81e8e36dd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:12.451 WARN 2458 --- [ocp-async-8,95d19bf98806cffc,f8e42599658d4ed5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:12.451 INFO 2458 --- [ocp-async-8,ed22473203c52ead,1d13f6ae2c7725da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:12.454 INFO 2458 --- [alarm-detect-10,62ec020a7b83acfa,7762cd9133db462b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761533985 ~ 1761534002 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534000, value=65.374)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534000, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534000=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:00:12.455 INFO 2458 --- [ocp-schedules-1,aa2820aa98fc913a,2edc71f6f7bd05e1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:00:12.455 INFO 2458 --- [ocp-schedules-1,aa2820aa98fc913a,2edc71f6f7bd05e1] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:00:12.463 INFO 2458 --- [ocp-schedules-1,aa2820aa98fc913a,2edc71f6f7bd05e1] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:00:12.463 INFO 2458 --- [ocp-schedules-1,aa2820aa98fc913a,2edc71f6f7bd05e1] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:00:13.011 INFO 2458 --- [ocp-schedules-14,84c00df23535f989,a3a290d6fa2eb2d9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:13.014 INFO 2458 --- [ocp-schedules-14,84c00df23535f989,a3a290d6fa2eb2d9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:13.299 INFO 2458 --- [ocp-schedules-24,e148d03a6315bc2f,4ff0d3fd82d81fe7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:00:13.299 INFO 2458 --- [ocp-schedules-24,e148d03a6315bc2f,4ff0d3fd82d81fe7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:00:13.361 INFO 2458 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761533940, nodeCount=817, affectRows=817 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions1,e148d03a6315bc2f,c9cd9b8425588b44] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions2,e148d03a6315bc2f,a1f24fbe5abb716a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions4,e148d03a6315bc2f,365e476ca46a962b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions3,e148d03a6315bc2f,856cc8a62809c66f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions6,e148d03a6315bc2f,741e4a91f29da06a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:00:13.374 INFO 2458 --- [Collect all tenants major compactions5,e148d03a6315bc2f,cfea3ce5701fafae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:00:13.410 INFO 2458 --- [Collect all tenants major compactions6,e148d03a6315bc2f,741e4a91f29da06a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.411 INFO 2458 --- [Collect all tenants major compactions5,e148d03a6315bc2f,cfea3ce5701fafae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.412 INFO 2458 --- [Collect all tenants major compactions5,e148d03a6315bc2f,cfea3ce5701fafae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:00:13.412 INFO 2458 --- [Collect all tenants major compactions6,e148d03a6315bc2f,741e4a91f29da06a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:00:13.417 INFO 2458 --- [Collect all tenants major compactions2,e148d03a6315bc2f,a1f24fbe5abb716a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.419 INFO 2458 --- [Collect all tenants major compactions2,e148d03a6315bc2f,a1f24fbe5abb716a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:00:13.421 INFO 2458 --- [Collect all tenants major compactions5,e148d03a6315bc2f,cfea3ce5701fafae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:00:13.423 INFO 2458 --- [Collect all tenants major compactions4,e148d03a6315bc2f,365e476ca46a962b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.423 INFO 2458 --- [Collect all tenants major compactions1,e148d03a6315bc2f,c9cd9b8425588b44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.423 INFO 2458 --- [Collect all tenants major compactions2,e148d03a6315bc2f,a1f24fbe5abb716a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:00:13.423 INFO 2458 --- [Collect all tenants major compactions6,e148d03a6315bc2f,741e4a91f29da06a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:00:13.424 INFO 2458 --- [Collect all tenants major compactions4,e148d03a6315bc2f,365e476ca46a962b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:00:13.425 INFO 2458 --- [Collect all tenants major compactions1,e148d03a6315bc2f,c9cd9b8425588b44] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:00:13.426 INFO 2458 --- [Collect all tenants major compactions3,e148d03a6315bc2f,856cc8a62809c66f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:13.427 INFO 2458 --- [Collect all tenants major compactions3,e148d03a6315bc2f,856cc8a62809c66f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:00:13.428 INFO 2458 --- [Collect all tenants major compactions4,e148d03a6315bc2f,365e476ca46a962b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:00:13.428 INFO 2458 --- [Collect all tenants major compactions1,e148d03a6315bc2f,c9cd9b8425588b44] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:00:13.430 INFO 2458 --- [Collect all tenants major compactions3,e148d03a6315bc2f,856cc8a62809c66f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:00:13.431 INFO 2458 --- [ocp-schedules-24,e148d03a6315bc2f,4ff0d3fd82d81fe7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:00:13.840 INFO 2458 --- [ocp-schedules-27,82fc9d99eabf98cf,480f062c2cf6c83e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:00:13.840+08:00, clusters=[1, 2] 2025-10-27 11:00:13.843 INFO 2458 --- [ocp-schedules-27,82fc9d99eabf98cf,480f062c2cf6c83e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:00:13.844 INFO 2458 --- [ocp-schedules-27,82fc9d99eabf98cf,480f062c2cf6c83e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:00:13.961 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=1000, affectRows=1000 2025-10-27 11:00:13.965 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=999, validCount=995 2025-10-27 11:00:13.965 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=1000, affectRows=1000 2025-10-27 11:00:13.970 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=994, validCount=992 2025-10-27 11:00:13.982 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835014129065861120-1761533880' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835014129065861120-1761533880' for key 'PRIMARY' 2025-10-27 11:00:14.033 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=1000, affectRows=1000 2025-10-27 11:00:14.038 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=992, validCount=988 2025-10-27 11:00:14.094 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=1000, affectRows=1000 2025-10-27 11:00:14.098 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=1000, validCount=996 2025-10-27 11:00:14.115 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=74, affectRows=74 2025-10-27 11:00:14.115 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=72, validCount=72 2025-10-27 11:00:14.451 INFO 2458 --- [ocp-async-3,05115e60b59b9785,af95cf86596a9347] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:14.450 WARN 2458 --- [ocp-async-6,b0116465bb39da5a,9faf1cd65a6f1ff4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:14.707 INFO 2458 --- [prometheus-minute-schedule-1,96b21f87557ac804,4982db49541aa67a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:14.727 INFO 2458 --- [prometheus-minute-schedule-1,96b21f87557ac804,4982db49541aa67a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:14.845 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:14.846 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:15.502 INFO 2458 --- [ocp-schedules-6,7beed39f9199a64a,244e9a0086200cad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:16.451 WARN 2458 --- [ocp-async-10,a5d604a38721925a,5bb91e959307474d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:16.451 INFO 2458 --- [ocp-async-7,0805465295f35a5a,70c659ae3963fed0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:17.707 INFO 2458 --- [prometheus-minute-schedule-1,98e2e13ae8d257aa,fd1d83cf599544d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:17.716 INFO 2458 --- [prometheus-minute-schedule-1,98e2e13ae8d257aa,fd1d83cf599544d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:17.791 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761533940, nodeCount=1000, affectRows=1000 2025-10-27 11:00:17.795 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761533940, nodeCount=1000, validCount=988 2025-10-27 11:00:17.846 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:17.847 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:18.451 WARN 2458 --- [ocp-async-7,82ec01ca47fbd8ed,0bd3ae90b7fea570] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:18.451 INFO 2458 --- [ocp-async-7,3f4d30a95da7cb94,7c9829de8ac38d7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:18.503 INFO 2458 --- [ocp-async-8,9aa4863b2669b07b,1e3ee1f7efd75601] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:18.745 INFO 2458 --- [ocp-schedules-12,380d34cad6c0465b,c1c349de40682d7e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:00:20.451 WARN 2458 --- [ocp-async-6,c908eef5d32d7481,7575a7072d21f072] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:20.451 INFO 2458 --- [ocp-async-1,ae46ea9d139b2c11,421b8332a42c7fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:20.707 INFO 2458 --- [prometheus-minute-schedule-1,b850e4358f7a959f,52d8f6bd308412a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:20.715 INFO 2458 --- [prometheus-minute-schedule-1,b850e4358f7a959f,52d8f6bd308412a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:20.847 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:20.848 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:21.410 INFO 2458 --- [ocp-schedules-23,e4d8b0fc0c42dab3,c6b0f06da5910d8c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [ocp-schedules-22,23cb2f5c33592f41,4bd2e8b22ba40cec] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:00:21.411 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:00:21.412 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:00:21.412 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:00:21.412 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:00:21.426 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:21.427 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:21.428 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:21.429 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:21.431 INFO 2458 --- [ocp-schedules-8,8ee4ef93d6d1e868,fe34d8bcbc4732d9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:21.431 INFO 2458 --- [host-check-clockdiff1,8ee4ef93d6d1e868,e413a09ea10bbf3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:00:21.431 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:21.432 INFO 2458 --- [host-check-clockdiff2,8ee4ef93d6d1e868,7bab2ff099d8121e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:00:21.432 INFO 2458 --- [host-check-clockdiff3,8ee4ef93d6d1e868,7f3846b36709cb03] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:00:21.433 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:21.433 INFO 2458 --- [host-check-clockdiff4,8ee4ef93d6d1e868,3211b19cfcea1571] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:00:21.433 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:21.433 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:21.434 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:21.434 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:21.434 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:21.434 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:21.434 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:21.437 INFO 2458 --- [host-check-clockdiff5,8ee4ef93d6d1e868,0d8dfc2b8c0191b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:00:21.439 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:21.443 INFO 2458 --- [ocp-schedules-1,aaa4cf35ddd61072,b08e5aabab822bfd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:00:21.457 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:21.463 INFO 2458 --- [host-check-clockdiff1,8ee4ef93d6d1e868,e413a09ea10bbf3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534021 0 0 , error= 2025-10-27 11:00:21.464 INFO 2458 --- [host-check-clockdiff5,8ee4ef93d6d1e868,0d8dfc2b8c0191b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534021 0 0 , error= 2025-10-27 11:00:21.465 INFO 2458 --- [host-check-clockdiff2,8ee4ef93d6d1e868,7bab2ff099d8121e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534021 -1 -1 , error= 2025-10-27 11:00:21.468 INFO 2458 --- [host-check-clockdiff4,8ee4ef93d6d1e868,3211b19cfcea1571] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534021 -1 -1 , error= 2025-10-27 11:00:21.504 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:00:21.505 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:00:21.505 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:00:21.506 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:00:21.506 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:00:21.506735255+08:00 2025-10-27 11:00:21.506 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:00:21.506146616+08:00 2025-10-27 11:00:21.507 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:00:21.507+08:00 2025-10-27 11:00:21.508 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:00:21.506+08:00 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status4,e4d8b0fc0c42dab3,fb4d5450976dff52] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:00:21.509 INFO 2458 --- [host-check-agent-status3,e4d8b0fc0c42dab3,e43f4a453d15c8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:00:21.511 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:00:21.514 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:00:21.514 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:00:21.514 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status1,e4d8b0fc0c42dab3,757b112a48c874be] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:00:21.506+08:00 and status AVAILABLE 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:00:21.507+08:00 and status AVAILABLE 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:00:21.519 INFO 2458 --- [host-check-agent-status5,e4d8b0fc0c42dab3,d4db81ed6f5abfa4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:00:21.520 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:00:21.520141064+08:00 2025-10-27 11:00:21.520 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:00:21.520 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:00:21.520+08:00 2025-10-27 11:00:21.521 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:00:21.520799204+08:00 2025-10-27 11:00:21.522 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:00:21.521+08:00 2025-10-27 11:00:21.523 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:00:21.523 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:00:21.523 INFO 2458 --- [host-check-agent-status2,e4d8b0fc0c42dab3,492973a101fb029f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:00:21.523 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:00:21.525 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:00:21.526 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:00:21.526 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:00:21.525455019+08:00 2025-10-27 11:00:21.527 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:00:21.526+08:00 2025-10-27 11:00:21.528 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:00:21.520+08:00 and status AVAILABLE 2025-10-27 11:00:21.529 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:00:21.521+08:00 and status AVAILABLE 2025-10-27 11:00:21.533 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:00:21.534 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:00:21.526+08:00 and status AVAILABLE 2025-10-27 11:00:21.535 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:00:21.538 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:00:21.543 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:21.545 INFO 2458 --- [ocp-schedules-6,8347b10cfbcd70e3,41193429da8b7bf2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:00:21.546 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:21.546 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:21.549 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:21.549 INFO 2458 --- [ocp-schedules-16,e43e4184fb3a5ee0,7db3efed1489de9a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:00:21.550 INFO 2458 --- [ocp-schedules-6,8347b10cfbcd70e3,41193429da8b7bf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:21.552 INFO 2458 --- [ocp-schedules-16,e43e4184fb3a5ee0,7db3efed1489de9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:21.553 INFO 2458 --- [ocp-schedules-16,e43e4184fb3a5ee0,7db3efed1489de9a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:00:21.553 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:21.553 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:21.554 INFO 2458 --- [ocp-schedules-4,b4d2ee3225e10743,f74949be4742a049] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:00:21.556 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:21.556 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:21.559 INFO 2458 --- [ocp-schedules-4,b4d2ee3225e10743,f74949be4742a049] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:21.561 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:21.563 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:21.568 INFO 2458 --- [ocp-schedules-24,a5a783cbb767e951,c15a338bf0b6ffb6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:00:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.604 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:21.605 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:00:21.610 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:21.611 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:21.614 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.618 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:21.619 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:21.619 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:21.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:00:21.623 INFO 2458 --- [ocp-schedules-6,8347b10cfbcd70e3,41193429da8b7bf2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:00:21.623 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.624 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:21.629 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:00:21.630 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:00:21.635 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:00:21.643 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:00:21.649 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:00:21.659 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:00:21.669 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:00:21.673 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:00:21.675 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:00:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.675 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:21.678 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:21.679 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.679 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:00:21.690 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:21.691 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:00:21.708 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:00:21.708 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:21.710 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:21.713 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:21.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:21.716974+08:00, lastProcessTime=2025-10-27T11:00:11.714022+08:00 2025-10-27 11:00:21.722 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:21.728 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:00:21.730 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:00:21.733 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:00:21.746 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:00:21.751 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:00:21.752 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:21.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.754 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:21.755 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.757 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:21.757 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.757 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.758 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.758 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.758 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:00:21.758 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:00:21.759 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:21.764 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:00:21.769 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:00:21.774 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:00:21.782 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:21.790 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:21.790 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:21.796 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:00:21.799 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:00:21.802 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:21.803 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:21.806 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:21.817 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:00:21.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:21.826 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:00:21.827 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:00:21.828 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:21.834 INFO 2458 --- [host-check-host-status5,23cb2f5c33592f41,418223414c0bc1c9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:00:21.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:21.850 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.851 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:21.852 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:00:21.854 INFO 2458 --- [host-check-host-status4,23cb2f5c33592f41,72ec8891d837fd49] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:00:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.856 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.856 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.857 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.857 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:21.858 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:21.858 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:21.858 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.859 INFO 2458 --- [host-check-host-status3,23cb2f5c33592f41,5a1c777449ae699c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:00:21.861 INFO 2458 --- [host-check-host-status2,23cb2f5c33592f41,b6e1aaffd3dffe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:00:21.862 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:00:21.863 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:00:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:21.871 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:00:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:21.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:21.892 INFO 2458 --- [host-check-host-status1,23cb2f5c33592f41,c4c8472cfdc92860] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:00:21.904 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:21.904 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:21.936 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:00:21.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:21.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:21.969 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:21.716974+08:00 2025-10-27 11:00:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:21.984 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:00:21.984 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.985 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:00:21.987 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:21.988 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:00:22.003 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:00:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:22.026 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:22.027 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:22.028 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:00:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:00:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:00:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:22.055 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:22.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:22.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:00:22.170 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:00:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:22.185 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:22.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:00:22.286 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:00:22.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.287 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:22.287 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:00:22.289 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.289 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.289 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:22.290 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:00:22.328 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:00:22.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:22.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.370 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:22.370 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:22.370 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:22.370 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:00:22.384 INFO 2458 --- [host-check-clockdiff3,8ee4ef93d6d1e868,7f3846b36709cb03] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534022 0 0 , error= 2025-10-27 11:00:22.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:00:22.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:00:22.415 INFO 2458 --- [ocp-schedules-4,b4d2ee3225e10743,f74949be4742a049] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:00:22.451 WARN 2458 --- [ocp-async-1,b3aa94f054fc75d4,89cf03b9d960b870] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:22.451 INFO 2458 --- [ocp-async-1,53812fabbb0d0376,2a5d127fa56386a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:22.464 INFO 2458 --- [ocp-schedules-26,1d36d2528ea9e31c,da64412e7095b6fd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:22.501 WARN 2458 --- [scanner-112114378-1,1d36d2528ea9e31c,d14af34b51de7b36] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:23.022 INFO 2458 --- [ocp-schedules-1,38240162af507322,1d5c4cc6322bb8ae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:23.025 INFO 2458 --- [ocp-schedules-1,38240162af507322,1d5c4cc6322bb8ae] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:23.707 INFO 2458 --- [prometheus-minute-schedule-1,c6ea41510f962a7f,5a19c14474190f81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:23.714 INFO 2458 --- [prometheus-minute-schedule-1,c6ea41510f962a7f,5a19c14474190f81] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:23.848 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:23.848 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:24.451 WARN 2458 --- [ocp-async-1,e7b5bf4a92bb0acb,45581170c3833426] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:24.451 INFO 2458 --- [ocp-async-5,c106c157ef5ad1bc,4035b1604c969a92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:25.510 INFO 2458 --- [ocp-schedules-27,3a425e11a1b154f2,51a9057157693b56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:26.451 WARN 2458 --- [ocp-async-5,377d99c313e91c84,12ae9cf67027b72b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:26.451 INFO 2458 --- [ocp-async-3,12382bc05ea4f4b0,959456736b178f1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:26.453 INFO 2458 --- [ocp-schedules-14,886ab8ea692b783a,264525dfef915f09] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:00:26.461 INFO 2458 --- [ocp-schedules-14,886ab8ea692b783a,264525dfef915f09] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:26.461 INFO 2458 --- [ocp-schedules-14,886ab8ea692b783a,264525dfef915f09] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:26.707 INFO 2458 --- [prometheus-minute-schedule-1,88765450475d7b52,80e8b07cf311f0a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:26.715 INFO 2458 --- [prometheus-minute-schedule-1,88765450475d7b52,80e8b07cf311f0a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:26.848 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:26.849 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:28.451 WARN 2458 --- [ocp-async-10,b6294c291ebc5773,52dd14722d5cb85a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:28.451 INFO 2458 --- [ocp-async-2,7a95289d00967538,222b44bb25c1747d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:28.503 INFO 2458 --- [ocp-async-1,bf734291e9f88d46,b860935823dd5048] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:28.770 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:00:28.770 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:00:28.771 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:00:28.771 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:00:28.773 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:00:28.773 INFO 2458 --- [ocp-schedules-6,cb198073168a2aba,3382d218832dc892] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:00:29.259 INFO 2458 --- [ocp-schedules-9,864b9c0c7fe976f9,17e0ee378a0f1d4d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:00:29.265 INFO 2458 --- [ocp-schedules-9,864b9c0c7fe976f9,17e0ee378a0f1d4d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:00:29.265 INFO 2458 --- [ocp-schedules-9,864b9c0c7fe976f9,17e0ee378a0f1d4d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:00:29.707 INFO 2458 --- [prometheus-minute-schedule-1,73cfc8af7bba56ea,ec5ee3f5b41eb6cc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:29.715 INFO 2458 --- [prometheus-minute-schedule-1,73cfc8af7bba56ea,ec5ee3f5b41eb6cc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:29.849 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:29.850 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:30.452 WARN 2458 --- [ocp-async-4,c9d611b9f98d8637,9345752459edca26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:30.452 INFO 2458 --- [ocp-async-7,f353db65755dd7e8,8660a75dad5d134f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:31.460 INFO 2458 --- [ocp-schedules-2,75a6f56ac01337b8,6d235ed4af97b689] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-10-27 11:00:31.460 INFO 2458 --- [ocp-schedules-2,75a6f56ac01337b8,6d235ed4af97b689] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-10-27 11:00:31.460 INFO 2458 --- [ocp-schedules-2,75a6f56ac01337b8,6d235ed4af97b689] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=8769 2025-10-27 11:00:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:31.715001+08:00, lastProcessTime=2025-10-27T11:00:21.716974+08:00 2025-10-27 11:00:31.731 INFO 2458 --- [ocp-schedules-2,75a6f56ac01337b8,6d235ed4af97b689] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=8769 2025-10-27 11:00:31.923 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:31.715001+08:00 2025-10-27 11:00:32.134 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2ed26f8bcb8ae29d,7186947145247eaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=2ed26f8bcb8ae29d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:00:32.136 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fa5e936f21901c1c,228fe446cd6d3663] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=fa5e936f21901c1c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:00:32.137 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0443eb8570372a0,713342733465577e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=e0443eb8570372a0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:00:32.138 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0443eb8570372a0,713342733465577e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=e0443eb8570372a0, duration=13 ms] 2025-10-27 11:00:32.162 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,916c3a2e95a5f350,58ebcdb174b2eec1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clustersView?page=1&size=10, client=10.40.198.13, traceId=916c3a2e95a5f350, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClustersView(String,Pageable), args=Page request [number: 0, size 10, sort: name: ASC],] 2025-10-27 11:00:32.163 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,339da9f3ad44f55b,6e3fd07a810f7110] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=339da9f3ad44f55b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:00:32.163 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,339da9f3ad44f55b,6e3fd07a810f7110] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=339da9f3ad44f55b, duration=38 ms] 2025-10-27 11:00:32.163 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2ed26f8bcb8ae29d,7186947145247eaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=2ed26f8bcb8ae29d, duration=38 ms] 2025-10-27 11:00:32.183 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,916c3a2e95a5f350,58ebcdb174b2eec1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clustersView client=10.40.198.13, traceId=916c3a2e95a5f350, duration=28 ms] 2025-10-27 11:00:32.284 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,d7fb06448115a967,c7723622f00fec60] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=d7fb06448115a967, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:00:32.300 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:00:32.300 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:00:32.300 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:00:32.305 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,d7fb06448115a967,c7723622f00fec60] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=d7fb06448115a967, duration=28 ms] 2025-10-27 11:00:32.381 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fa5e936f21901c1c,228fe446cd6d3663] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=fa5e936f21901c1c, duration=256 ms] 2025-10-27 11:00:32.452 WARN 2458 --- [ocp-async-7,4f2f1c655eaee231,6a935d9555ad0ccd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:32.452 INFO 2458 --- [ocp-async-8,84c3978d6c4bd344,396f32166ab0b128] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:32.648 INFO 2458 --- [ocp-schedules-27,f5559195e50be8e5,73f8b1e30d5d77bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:32.707 INFO 2458 --- [prometheus-minute-schedule-1,fbe15d048249ce50,d53c0b52d2275cc8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:32.717 INFO 2458 --- [prometheus-minute-schedule-1,fbe15d048249ce50,d53c0b52d2275cc8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:32.746 WARN 2458 --- [scanner-112114686-1,f5559195e50be8e5,f7088c55ff8f1f1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:32.814 INFO 2458 --- [alarm-detect-1,f5559195e50be8e5,c336edfacd27a798] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534000 ~ 1761534022 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534015, value=83.747)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534015, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534015=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:00:32.850 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:32.851 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:33.033 INFO 2458 --- [ocp-schedules-21,468294f7ca769c62,e195e3312156fa73] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:33.037 INFO 2458 --- [ocp-schedules-21,468294f7ca769c62,e195e3312156fa73] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:33.543 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:00:33.543 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:00:33.609 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:33.609 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:33.650 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:33.650 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:33.651 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:33.652 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:00:33.665 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:33.665 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:33.665 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:33.666 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:00:33.673 INFO 2458 --- [ocp-schedules-7,b62687aaa751eced,02e0711df8070b5a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:00:33.673 INFO 2458 --- [ocp-schedules-7,b62687aaa751eced,02e0711df8070b5a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:00:33.728 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:33.730 INFO 2458 --- [ocp-schedules-26,49d83153e401ad7b,6ffc1d6f7b643372] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:00:33.810 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:00:33.810 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:00:33.829 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:00:33.829 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:00:33.829 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:00:33.845 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:00:33.845 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:00:33.845 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:00:33.857 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:00:33.857 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:00:33.857 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:00:33.869 INFO 2458 --- [ocp-schedules-29,fe38eaa3fb838601,0e86173d6ce5558a] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:00:34.123 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,67aa5a0abb85383d,a08e010463ab29c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=67aa5a0abb85383d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:00:34.123 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,67aa5a0abb85383d,a08e010463ab29c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=67aa5a0abb85383d, duration=9 ms] 2025-10-27 11:00:34.126 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9f59905346465e0f,d88779bbd3dd6047] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=9f59905346465e0f, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:00:34.126 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9f59905346465e0f,d88779bbd3dd6047] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=9f59905346465e0f, duration=12 ms] 2025-10-27 11:00:34.127 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,a12950c4a5d676f9,e1346165bf046141] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=a12950c4a5d676f9, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:00:34.127 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,a12950c4a5d676f9,e1346165bf046141] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=a12950c4a5d676f9, duration=12 ms] 2025-10-27 11:00:34.152 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:00:34.152 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:00:34.153 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:00:34.452 INFO 2458 --- [ocp-async-3,351a33794f990ba3,ce12bb6e8e68bef8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:34.452 WARN 2458 --- [ocp-async-9,e6d539333ff6c208,0962fab0de8d3d2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:35.516 INFO 2458 --- [ocp-schedules-6,8f09a7fa31632cb1,75e16c8769445384] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:35.707 INFO 2458 --- [prometheus-minute-schedule-1,faf61f537858e791,a383fc22e9e76ba2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:35.714 INFO 2458 --- [prometheus-minute-schedule-1,faf61f537858e791,a383fc22e9e76ba2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:35.851 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:35.852 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:36.239 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:36.239 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:00:36.239 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:00:36.240 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:00:36.240 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:00:36.240 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:00:36.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:36.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:36.253 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:36.253 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:36.257 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:36.257 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:36.257 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:36.257 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:36.258 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:36.261 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:36.317 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:00:36.317 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:00:36.317 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:00:36.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:00:36.324 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:00:36.326 INFO 2458 --- [ocp-schedules-8,e69f7742204d9a04,a5513271ffb9fdd5] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:00:36.452 WARN 2458 --- [ocp-async-2,329cfa98a9202be7,d94018169f2bb7d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:36.452 INFO 2458 --- [ocp-async-4,f16b5ca83df545d2,cb12e881e964e86c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:37.112 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,d43f5ca7b53a83fb,d6243784cbabffa0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=d43f5ca7b53a83fb, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:00:37.117 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,d43f5ca7b53a83fb,d6243784cbabffa0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=d43f5ca7b53a83fb, duration=13 ms] 2025-10-27 11:00:37.708 INFO 2458 --- [prometheus-cache-monitor1,3149e44a9974d01b,e825b315849e05ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20278,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5075,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":9830} 2025-10-27 11:00:37.708 INFO 2458 --- [prometheus-cache-monitor1,3149e44a9974d01b,e825b315849e05ee] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:00:38.452 INFO 2458 --- [ocp-async-7,59e7900729c81d22,b4501cbc355a67e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:38.452 WARN 2458 --- [ocp-async-6,4d11da59e89a6faa,dc58faf187c98114] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:38.503 INFO 2458 --- [ocp-async-9,e390f70e3e2c91d5,19df02bb6f31b461] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:38.539 INFO 2458 --- [ocp-schedules-15,0db8f42150b30eba,1677d7c3bed9d210] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:00:38.543 INFO 2458 --- [ocp-schedules-15,0db8f42150b30eba,1677d7c3bed9d210] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:38.544 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:00:38.544 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:00:38.544 INFO 2458 --- [ocp-schedules-15,0db8f42150b30eba,1677d7c3bed9d210] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:00:38.707 INFO 2458 --- [prometheus-minute-schedule-1,6c6f51f71ccc5079,1537d129fe2c9760] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:38.715 INFO 2458 --- [prometheus-minute-schedule-1,6c6f51f71ccc5079,1537d129fe2c9760] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:38.852 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:38.853 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:40.009 INFO 2458 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:00:40.010 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:00:40.047 INFO 2458 --- [Check obproxy1,fd4bace9ee4ebf9b,e765ca0c0e39bfa0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:00:40.047 INFO 2458 --- [Check obproxy1,fd4bace9ee4ebf9b,e765ca0c0e39bfa0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:00:40.048 INFO 2458 --- [Collect obproxy1,8da889ad0a40d2b8,e002e60e403c90c0] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:00:40.048 INFO 2458 --- [Collect obproxy1,8da889ad0a40d2b8,e002e60e403c90c0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:00:40.079 INFO 2458 --- [ocp-schedules-15,30b05a2295df0039,a090afaf06cea66e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:00:40.079 INFO 2458 --- [ocp-schedules-15,30b05a2295df0039,a090afaf06cea66e] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:00:40.083 INFO 2458 --- [Check obproxy1,fd4bace9ee4ebf9b,e765ca0c0e39bfa0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:00:40.083 INFO 2458 --- [Collect obproxy1,8da889ad0a40d2b8,e002e60e403c90c0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:00:40.086 INFO 2458 --- [Collect obproxy1,8da889ad0a40d2b8,e002e60e403c90c0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:00:40.087 INFO 2458 --- [Check obproxy1,fd4bace9ee4ebf9b,e765ca0c0e39bfa0] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:00:40.087 INFO 2458 --- [Check obproxy1,fd4bace9ee4ebf9b,e765ca0c0e39bfa0] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:00:40.089 INFO 2458 --- [Collect obproxy1,8da889ad0a40d2b8,e002e60e403c90c0] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:00:40.452 INFO 2458 --- [ocp-async-9,4c29b235d1a57726,526b099eb886a831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:40.452 WARN 2458 --- [ocp-async-3,f588823c92c1da03,f86d98e1621473b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:41.707 INFO 2458 --- [prometheus-minute-schedule-1,559383f086f6eb96,5743988c2bed6845] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:41.713 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:41.712768+08:00, lastProcessTime=2025-10-27T11:00:31.715001+08:00 2025-10-27 11:00:41.715 INFO 2458 --- [prometheus-minute-schedule-1,559383f086f6eb96,5743988c2bed6845] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:41.853 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:41.853 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:41.920 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:41.712768+08:00 2025-10-27 11:00:42.452 INFO 2458 --- [ocp-async-1,3ae8eb775e237315,8bf5adafb393a29f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:42.453 WARN 2458 --- [ocp-async-4,c8c33623c5103df0,b88912dceb91a52a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:42.608 INFO 2458 --- [ocp-schedules-23,0258bf72f065f2d4,9182c2ce2dc07943] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:00:42.608 INFO 2458 --- [ocp-schedules-23,0258bf72f065f2d4,9182c2ce2dc07943] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:00:42.615 INFO 2458 --- [ocp-schedules-23,0258bf72f065f2d4,9182c2ce2dc07943] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:00:42.615 INFO 2458 --- [ocp-schedules-23,0258bf72f065f2d4,9182c2ce2dc07943] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:00:42.821 INFO 2458 --- [ocp-schedules-26,5540954d72f31d8a,2604ccba268f2d55] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:42.861 WARN 2458 --- [scanner-112114840-1,5540954d72f31d8a,9b395678c5db4916] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:43.025 INFO 2458 --- [alarm-detect-7,5540954d72f31d8a,65fd7c23fcc1b3d5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534015 ~ 1761534033 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534030, value=79.151)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534030, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534030=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:00:43.045 INFO 2458 --- [ocp-schedules-8,5b24fde925e9a7d3,684aff772863bacd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:43.050 INFO 2458 --- [ocp-schedules-8,5b24fde925e9a7d3,684aff772863bacd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:43.910 INFO 2458 --- [ocp-schedules-30,9f60c1b2f213ea46,df86d89ea01be059] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:00:44.453 INFO 2458 --- [ocp-async-6,10c8436c9df6bc32,dc6549eb55beae51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:44.453 WARN 2458 --- [ocp-async-7,9061022283629b12,9bd8907e5f7e935d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:44.707 INFO 2458 --- [prometheus-minute-schedule-1,4b4baf387b5a359f,ca65abeae564294b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:44.715 INFO 2458 --- [prometheus-minute-schedule-1,4b4baf387b5a359f,ca65abeae564294b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:44.853 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:44.854 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:45.102 INFO 2458 --- [prometheus-second-schedule-1,d975b753900c1a42,7b8e0669395e6c50] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:00:45.524 INFO 2458 --- [ocp-schedules-15,739737f6522dd8dc,b25ab99025af6d9b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:46.453 INFO 2458 --- [ocp-async-3,252a1eb44f5ebb5f,c72345de8e6db048] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:46.453 WARN 2458 --- [ocp-async-10,a52930328d89d9cf,d8fb893601991976] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:46.463 INFO 2458 --- [ocp-schedules-19,3a5f170d6b2406cc,5165fa0c589c377d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:00:46.471 INFO 2458 --- [ocp-schedules-19,3a5f170d6b2406cc,5165fa0c589c377d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:46.471 INFO 2458 --- [ocp-schedules-19,3a5f170d6b2406cc,5165fa0c589c377d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:00:46.856 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2136550c519bbfb7,fc4e7c9f8f70b99e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=2136550c519bbfb7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:00:46.856 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2136550c519bbfb7,fc4e7c9f8f70b99e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=2136550c519bbfb7, duration=1 ms] 2025-10-27 11:00:47.158 INFO 2458 --- [ocp-schedules-10,444416d2da2cf6f0,050f508e1765cbde] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:00:47.158+08:00, clusters=[1, 2] 2025-10-27 11:00:47.175 INFO 2458 --- [ocp-schedules-10,444416d2da2cf6f0,050f508e1765cbde] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:00:47.178 INFO 2458 --- [ocp-schedules-10,444416d2da2cf6f0,050f508e1765cbde] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:00:47.707 INFO 2458 --- [prometheus-minute-schedule-1,6d20b7724fd24dca,3eacb9f9e057969e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:47.714 INFO 2458 --- [prometheus-minute-schedule-1,6d20b7724fd24dca,3eacb9f9e057969e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:47.854 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:47.855 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:48.453 INFO 2458 --- [ocp-async-4,a002b5ee3dd3601b,21d9403c7301b727] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:48.453 WARN 2458 --- [ocp-async-2,c3f2f875297bc2b2,b3b9a20fc57b72d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:48.503 INFO 2458 --- [ocp-async-1,296e32a4d481d832,de2790495892aa55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:50.453 WARN 2458 --- [ocp-async-6,c8b699feda44b2ff,2831ac35deccc3ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:50.453 INFO 2458 --- [ocp-async-5,1e26d6e770acfd91,34d849e10fad1d81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:50.707 INFO 2458 --- [prometheus-minute-schedule-1,5d38152ea2b7d9bc,dd36b7b7cdbbb04d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:50.714 INFO 2458 --- [prometheus-minute-schedule-1,5d38152ea2b7d9bc,dd36b7b7cdbbb04d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:50.855 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:50.856 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:51.410 INFO 2458 --- [ocp-schedules-22,c9dfcfd3a9b4e318,10e7b3ffb88a13b7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:51.410 INFO 2458 --- [ocp-schedules-11,b7d951163f410a37,df4197f4f5cfc888] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:51.410 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:00:51.410 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:00:51.410 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:00:51.411 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:00:51.411 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:00:51.411 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:00:51.411 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:00:51.412 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:00:51.412 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:00:51.412 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:00:51.442 INFO 2458 --- [ocp-schedules-26,52d290f6190b5049,5292fcbd3be9b114] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:00:51.445 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:51.445 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:51.446 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:51.446 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:51.446 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:51.447 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:51.447 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:51.450 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:51.450 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:51.451 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:51.451 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:51.451 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:51.452 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:51.452 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:51.452 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:51.455 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:51.455 INFO 2458 --- [ocp-schedules-14,92f0cb8d57e45671,bc4dac8b223c9864] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:00:51.455 INFO 2458 --- [host-check-clockdiff1,92f0cb8d57e45671,1e2cba68208eadbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:00:51.455 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:51.456 INFO 2458 --- [host-check-clockdiff4,92f0cb8d57e45671,f7012929fccda146] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:00:51.456 INFO 2458 --- [host-check-clockdiff2,92f0cb8d57e45671,99bfed03302ec083] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:00:51.456 INFO 2458 --- [host-check-clockdiff5,92f0cb8d57e45671,0b0058992d4787b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:00:51.456 INFO 2458 --- [host-check-clockdiff3,92f0cb8d57e45671,2bca841ecf811957] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:00:51.456 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:51.458 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:51.460 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:51.491 INFO 2458 --- [host-check-clockdiff1,92f0cb8d57e45671,1e2cba68208eadbe] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534051 0 0 , error= 2025-10-27 11:00:51.491 INFO 2458 --- [host-check-clockdiff2,92f0cb8d57e45671,99bfed03302ec083] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534051 -1 -1 , error= 2025-10-27 11:00:51.495 INFO 2458 --- [host-check-clockdiff3,92f0cb8d57e45671,2bca841ecf811957] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534051 -1 -1 , error= 2025-10-27 11:00:51.496 INFO 2458 --- [host-check-clockdiff4,92f0cb8d57e45671,f7012929fccda146] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534051 -1 -1 , error= 2025-10-27 11:00:51.530 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:00:51.530 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:00:51.530 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:00:51.531 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:00:51.532 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:00:51.531192770+08:00 2025-10-27 11:00:51.532 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status5,c9dfcfd3a9b4e318,20ebe9d988ac77b3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status3,c9dfcfd3a9b4e318,29c3a0e571da5a2a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:00:51.536 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:00:51.537 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:00:51.536836219+08:00 2025-10-27 11:00:51.537 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:00:51.537+08:00 2025-10-27 11:00:51.539 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:00:51.532+08:00 2025-10-27 11:00:51.539 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:00:51.539330177+08:00 2025-10-27 11:00:51.540 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:00:51.535 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:00:51.541 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:00:51.541 INFO 2458 --- [host-check-agent-status4,c9dfcfd3a9b4e318,9b4485345aa663ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:00:51.542 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:00:51.539+08:00 2025-10-27 11:00:51.542 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:00:51.542 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:00:51.541595792+08:00 2025-10-27 11:00:51.543 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:00:51.542+08:00 2025-10-27 11:00:51.545 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:00:51.545 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:00:51.545 INFO 2458 --- [host-check-agent-status2,c9dfcfd3a9b4e318,5e8054c8c60c8d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:00:51.545 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:00:51.545 INFO 2458 --- [ocp-schedules-23,fd1ee8bffdddb405,7b1e773717bddcc4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:00:51.548 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:00:51.532+08:00 and status AVAILABLE 2025-10-27 11:00:51.548 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:00:51.548 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:00:51.548 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:00:51.537+08:00 and status AVAILABLE 2025-10-27 11:00:51.548 INFO 2458 --- [host-check-agent-status1,c9dfcfd3a9b4e318,284a64353f927c1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:00:51.549 INFO 2458 --- [ocp-schedules-28,939e4e3c93512339,a283b69160c6db11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:00:51.550 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:00:51.551 INFO 2458 --- [ocp-schedules-23,fd1ee8bffdddb405,7b1e773717bddcc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:51.553 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:00:51.553 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:00:51.550774196+08:00 2025-10-27 11:00:51.554 INFO 2458 --- [ocp-schedules-24,58a844cd14935110,a84200f95c4dfcd5] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:00:51.554 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:00:51.542+08:00 and status AVAILABLE 2025-10-27 11:00:51.554 INFO 2458 --- [ocp-schedules-28,939e4e3c93512339,a283b69160c6db11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:51.554 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:00:51.553+08:00 2025-10-27 11:00:51.555 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:00:51.539+08:00 and status AVAILABLE 2025-10-27 11:00:51.556 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:00:51.558 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:00:51.558 INFO 2458 --- [ocp-schedules-24,58a844cd14935110,a84200f95c4dfcd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:51.560 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:00:51.564 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:00:51.553+08:00 and status AVAILABLE 2025-10-27 11:00:51.565 INFO 2458 --- [ocp-schedules-28,939e4e3c93512339,a283b69160c6db11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:00:51.566 INFO 2458 --- [ocp-schedules-10,8d9e91c157a351d4,fb6d094839ea45ef] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:00:51.568 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:00:51.568 INFO 2458 --- [ocp-schedules-8,02a77e56ff170451,fcf2d51274b48ea4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:00:51.569 INFO 2458 --- [ocp-schedules-19,95279e62757b9e6b,5e2fee0c99386c4e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:00:51.576 INFO 2458 --- [ocp-schedules-8,02a77e56ff170451,fcf2d51274b48ea4] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:00:51.581 INFO 2458 --- [ocp-schedules-19,95279e62757b9e6b,5e2fee0c99386c4e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:00:51.594 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:51.597 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:51.606 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:00:51.611 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:00:51.613 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:00:51.616 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:00:51.617 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:00:51.619 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:00:51.623 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:51.626 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.626 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:51.626 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:51.628 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:00:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.640 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:51.646 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:00:51.647 INFO 2458 --- [ocp-schedules-23,fd1ee8bffdddb405,7b1e773717bddcc4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:00:51.665 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:51.669 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.670 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:51.679 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:51.681 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:51.686 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:51.686 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:00:51.686 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:00:51.694 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:00:51.704 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:00:51.707 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:00:51.709 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.709 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.710 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:00:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.711 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:51.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:00:51.713578+08:00, lastProcessTime=2025-10-27T11:00:41.712768+08:00 2025-10-27 11:00:51.733 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:00:51.744 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:00:51.746 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:51.747 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.747 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:00:51.749 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:00:51.749 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:00:51.757 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:00:51.764 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:51.781 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:51.782 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:51.782 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:51.783 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:00:51.793 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:00:51.797 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:00:51.799 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:00:51.804 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:51.804 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.804 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.804 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.804 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.805 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.805 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:00:51.812 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:00:51.815 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:00:51.818 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:00:51.843 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:51.843 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:00:51.843 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.844 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:51.845 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.847 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:51.848 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:00:51.848 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:00:51.848 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:00:51.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:51.857 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:00:51.868 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:00:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:51.876 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:51.883 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:51.883 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:51.886 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:00:51.892 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:00:51.895 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:51.899 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:00:51.899 INFO 2458 --- [host-check-host-status2,b7d951163f410a37,6f810878080892c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:00:51.907 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:51.907 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.907 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.908 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.908 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.908 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:00:51.908 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.908 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:00:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:00:51.915 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:00:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:51.928 INFO 2458 --- [host-check-host-status4,b7d951163f410a37,bb31f33cfa6f0805] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:00:51.932 INFO 2458 --- [host-check-host-status5,b7d951163f410a37,ceb8a72cf667b091] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:00:51.936 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:51.936 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:51.937 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:51.938 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:00:51.941 INFO 2458 --- [host-check-host-status3,b7d951163f410a37,7434ccbfde66aa09] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:00:51.955 INFO 2458 --- [host-check-host-status1,b7d951163f410a37,e9f5ecc4ec9fad38] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:00:51.959 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:00:51.969 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:00:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:51.972 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:00:51.713578+08:00 2025-10-27 11:00:51.973 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:00:52.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:52.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:52.031 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:00:52.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:52.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:52.073 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:00:52.074 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.074 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:00:52.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.077 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:00:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:00:52.094 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:00:52.133 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:52.134 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.134 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:00:52.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:52.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:00:52.164 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:00:52.164 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:00:52.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:00:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.219 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.219 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:00:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:00:52.232 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:00:52.240 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:00:52.243 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:00:52.297 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:00:52.345 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:00:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:00:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:00:52.383 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:00:52.403 INFO 2458 --- [host-check-clockdiff5,92f0cb8d57e45671,0b0058992d4787b7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534052 0 0 , error= 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:00:52.453 WARN 2458 --- [ocp-async-2,cb18fe3cdd0bc73d,820b894ea7b165e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:52.453 INFO 2458 --- [ocp-async-2,f4b7cc18aa0b4a7c,078217cfdc232b3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:52.469 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:00:52.469 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:00:52.469 INFO 2458 --- [ocp-schedules-24,58a844cd14935110,a84200f95c4dfcd5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:00:53.031 INFO 2458 --- [ocp-schedules-19,d4470e15d46210ca,4196a69e8872a753] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:00:53.058 INFO 2458 --- [ocp-schedules-30,a4d52b15603cb5b9,777f2be79136930b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:00:53.062 INFO 2458 --- [ocp-schedules-30,a4d52b15603cb5b9,777f2be79136930b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:00:53.066 WARN 2458 --- [scanner-112115103-1,d4470e15d46210ca,197ade67332759b3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:00:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:00:53.393 INFO 2458 --- [ocp-schedules-18,04ea6a8ffce089bd,161fd0b0401c7ab6] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:00:53.399 INFO 2458 --- [ocp-schedules-18,04ea6a8ffce089bd,161fd0b0401c7ab6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:00:53.480 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:00:53.481 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.484 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:00:53.493 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.494 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.519 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.519 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.541 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:00:53.543 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.543 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.545 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.546 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.557 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.557 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.557 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.557 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.571 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.572 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.584 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.584 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.585 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.585 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.599 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.600 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.612 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.612 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.612 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.612 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:00:53.615 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.615 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.615 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.615 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.632 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.633 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.634 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.635 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.635 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.636 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.637 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.637 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.638 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.639 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.641 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:00:53.642 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.642 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.642 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.643 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.643 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:00:53.671 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.671 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.672 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.672 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:00:53.676 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.676 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.677 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:00:53.707 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:53.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.707 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.707 INFO 2458 --- [prometheus-minute-schedule-1,a66dc09ebe18d360,74ae29127a384b89] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:53.708 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.708 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:00:53.708 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:00:53.715 INFO 2458 --- [prometheus-minute-schedule-1,a66dc09ebe18d360,74ae29127a384b89] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:53.719 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.719 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:00:53.719 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.719 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:00:53.723 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:00:53.723 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:00:53.744 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.744 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.744 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.745 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:00:53.747 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.747 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.747 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.748 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:00:53.778 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.778 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.778 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.779 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:00:53.810 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.810 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.811 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.811 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:00:53.856 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:53.857 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:53.863 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.863 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.863 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.864 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.865 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.865 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.865 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.866 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.866 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.866 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.866 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.867 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.867 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.867 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.867 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.868 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.869 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.869 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.869 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.870 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.870 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.870 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.870 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.871 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.871 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.871 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.872 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.873 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.873 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.873 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.873 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.874 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.874 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.874 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.876 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.876 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.876 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.877 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:00:53.877 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.877 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.877 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.878 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:00:53.878 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.878 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.878 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.879 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:00:53.880 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:00:53.882 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:00:53.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:00:53.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:00:53.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:00:53.964 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:00:53.964 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:00:53.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:00:53.993 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:00:53.993 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:00:53.993 INFO 2458 --- [ocp-schedules-18,04ea6a8ffce089bd,161fd0b0401c7ab6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:00:54.104 INFO 2458 --- [ocp-schedules-17,8555381c88477fd5,e273632e4dd08d44] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:00:54.453 WARN 2458 --- [ocp-async-6,7533b2e4228eae1d,84ba34b8e848b31b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:54.453 INFO 2458 --- [ocp-async-7,2f9c693c0eb99d4c,f85fc661d625a081] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:55.530 INFO 2458 --- [ocp-schedules-18,e609503d7d978c45,46d65b1ab1826b86] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:00:56.453 INFO 2458 --- [ocp-async-1,0634c64fdd3f37cf,b4b488dab39fc9df] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:56.453 WARN 2458 --- [ocp-async-10,a68d8bf7339216bd,2156fab37d2921fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:56.708 INFO 2458 --- [prometheus-minute-schedule-1,e54c0c6332992c4a,3a8cbdf7c9ee99e2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:56.714 INFO 2458 --- [prometheus-minute-schedule-1,e54c0c6332992c4a,3a8cbdf7c9ee99e2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:56.857 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:56.858 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:00:57.567 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,1eea3f9938c07266,305452f5a2615c56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=1eea3f9938c07266, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:00:57.567 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,02071a9a3335f70d,f9248f4ffdb0a2f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters?null, client=10.40.198.13, traceId=02071a9a3335f70d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClusters(Pageable), args=Page request [number: 0, size 5000, sort: name: ASC],] 2025-10-27 11:00:57.567 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,1eea3f9938c07266,305452f5a2615c56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=1eea3f9938c07266, duration=9 ms] 2025-10-27 11:00:57.569 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,447d672189dbfbdd,d9126014a23f7938] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=447d672189dbfbdd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:00:57.571 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6a238a875b3f08d5,370126a14ef5e57f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=6a238a875b3f08d5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:00:57.573 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6df4f91872ff7da4,ee066c4af4e303f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=6df4f91872ff7da4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 11:00:57.575 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,447d672189dbfbdd,d9126014a23f7938] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=447d672189dbfbdd, duration=17 ms] 2025-10-27 11:00:57.587 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6a238a875b3f08d5,370126a14ef5e57f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=6a238a875b3f08d5, duration=29 ms] 2025-10-27 11:00:57.595 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,02071a9a3335f70d,f9248f4ffdb0a2f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters client=10.40.198.13, traceId=02071a9a3335f70d, duration=37 ms] 2025-10-27 11:00:57.596 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6df4f91872ff7da4,ee066c4af4e303f1] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:00:57.597 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6df4f91872ff7da4,ee066c4af4e303f1] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:00:57.608 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,bb2a6ba7ba3b13c0,87aded973a738f52] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=bb2a6ba7ba3b13c0, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:00:57.608 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6538c404b439af8e,7a77877310082174] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=6538c404b439af8e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:00:57.609 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,bb2a6ba7ba3b13c0,87aded973a738f52] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=bb2a6ba7ba3b13c0, duration=8 ms] 2025-10-27 11:00:57.648 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,684b66155a41acd8,299be80c5781c569] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=684b66155a41acd8, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:00:57.668 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,684b66155a41acd8,299be80c5781c569] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=684b66155a41acd8, duration=27 ms] 2025-10-27 11:00:57.724 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6df4f91872ff7da4,ee066c4af4e303f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=6df4f91872ff7da4, duration=166 ms] 2025-10-27 11:00:57.751 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:00:57.752 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-3,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:00:57.752 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:00:57.759 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3d56a4ce6dd78cdb,f0f18b251766e5eb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=3d56a4ce6dd78cdb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 11:00:57.776 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3d56a4ce6dd78cdb,f0f18b251766e5eb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:00:57.776 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3d56a4ce6dd78cdb,f0f18b251766e5eb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:00:57.840 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6538c404b439af8e,7a77877310082174] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=6538c404b439af8e, duration=239 ms] 2025-10-27 11:00:57.902 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3d56a4ce6dd78cdb,f0f18b251766e5eb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=3d56a4ce6dd78cdb, duration=150 ms] 2025-10-27 11:00:58.454 INFO 2458 --- [ocp-async-3,a6f1fbd69e3fdecf,80dedb59103d8884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:00:58.454 WARN 2458 --- [ocp-async-8,f16100478420ecbf,073153d00581ee19] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:00:58.503 INFO 2458 --- [ocp-async-2,55135fec5b235962,f16f2f2224584684] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:00:59.056 INFO 2458 --- [ocp-schedules-23,cdd0c82a8aafa20a,8c7f413f018b4e02] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:00:59.558 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,8f5950fdf4903d30,d73609cc9d81b207] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=8f5950fdf4903d30, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:00:59.558 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,8f5950fdf4903d30,d73609cc9d81b207] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=8f5950fdf4903d30, duration=8 ms] 2025-10-27 11:00:59.564 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7cc981e9b9040d8c,a1e1b2c0d28f1cd1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=7cc981e9b9040d8c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:00:59.564 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7cc981e9b9040d8c,a1e1b2c0d28f1cd1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=7cc981e9b9040d8c, duration=16 ms] 2025-10-27 11:00:59.565 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,ade7b774740b780c,1ec19f1dcc39400f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=ade7b774740b780c, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:00:59.565 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,ade7b774740b780c,1ec19f1dcc39400f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=ade7b774740b780c, duration=15 ms] 2025-10-27 11:00:59.570 INFO 2458 --- [ocp-schedules-14,d34abefd6ff7ee4c,aef5f8bcb1afc26a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:00:59.570 INFO 2458 --- [ocp-schedules-14,d34abefd6ff7ee4c,aef5f8bcb1afc26a] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:00:59.586 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:00:59.586 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:00:59.586 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:00:59.707 INFO 2458 --- [prometheus-minute-schedule-1,f3fccf3e5efdcdb4,bddc2a8b085efd9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534060, interval=60 2025-10-27 11:00:59.715 INFO 2458 --- [prometheus-minute-schedule-1,f3fccf3e5efdcdb4,bddc2a8b085efd9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:00:59.858 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:00:59.859 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:00.170 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 10:01:01.0, nextRun=2025-10-27 11:01:00.0, currentDate=Mon Oct 27 11:01:00 CST 2025 2025-10-27 11:01:00.170 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999995, name=Collect cluster snapshot, templateName=CollectClusterSnapshotSchedule, scheduleRule=0 1 * * * *, lastRunTime=2025-10-27 10:01:01.0, nextRunTime=2025-10-27 11:01:00.0} 2025-10-27 11:01:00.172 WARN 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2025-10-27 11:01:00.172 WARN 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] o.p.s.d.i.CollectClusterSnapshotSchedule : Snapshot service is disabled. Please check config: ocp.perf.snapshot.generation.enable 2025-10-27 11:01:00.173 WARN 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] c.o.o.c.t.e.c.worker.ReadyTaskWorker : failed to get template for class, action=com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule java.lang.NullPointerException: null at com.oceanbase.ocp.perf.sql.diag.internal.CollectClusterSnapshotSchedule.ready(CollectClusterSnapshotSchedule.java:74) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.getTaskMeta(ReadyTaskWorker.java:373) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.checkAndScheduleCronTask(ReadyTaskWorker.java:178) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.lambda$loopScheduleTask$1(ReadyTaskWorker.java:139) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.core.task.engine.coordinator.worker.AbstractTaskWorker.transactionWithoutResult(AbstractTaskWorker.java:61) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.loopScheduleTask(ReadyTaskWorker.java:138) at com.oceanbase.ocp.core.task.engine.coordinator.worker.ReadyTaskWorker.work(ReadyTaskWorker.java:107) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lockAndExecute(TaskCoordinator.java:136) at com.oceanbase.ocp.core.task.engine.coordinator.TaskCoordinator.lambda$startup$0(TaskCoordinator.java:74) at com.oceanbase.ocp.common.trace.TraceDecorator.lambda$decorate$0(TraceDecorator.java:33) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-10-27 11:01:00.179 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] c.o.o.c.t.e.c.worker.SubtaskGenerator : Split list is empty, name=Collect cluster hour metric, splitKey=cluster_ids, context=Argument{parallelIdx=-1, stringMap={}, listMap={}} 2025-10-27 11:01:00.183 WARN 2458 --- [task-coordinator3,8964d58b78e8b300,4fc4b94384e111eb] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Subtask is empty, mark task failed, name=Collect cluster hour metric, taskId=9093 2025-10-27 11:01:00.454 INFO 2458 --- [ocp-async-4,032c3949b1a7345a,c7550fe74eea4467] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:00.454 WARN 2458 --- [ocp-async-7,eac7699364617088,0d47d500a3be46b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:01.524 INFO 2458 --- [ocp-schedules-7,cde57783290d1040,9bce183312a37635] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:01:01.531 INFO 2458 --- [ocp-schedules-7,cde57783290d1040,9bce183312a37635] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:01.580 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:01.580 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:01:01.593 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:01.593 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:01.594 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:01.594 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:01.594 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:01.594 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:01:01.607 INFO 2458 --- [ocp-schedules-7,cde57783290d1040,9bce183312a37635] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:01:01.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:01.716491+08:00, lastProcessTime=2025-10-27T11:00:51.713578+08:00 2025-10-27 11:01:01.940 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:01.716491+08:00 2025-10-27 11:01:02.454 WARN 2458 --- [ocp-async-5,2c925905897bc8db,0c8161b7ad938790] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:02.454 INFO 2458 --- [ocp-async-10,a00ec2146818daad,a76b591f84450fb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:02.561 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,d79c346d1b02863e,3092c2ff77d65b45] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=d79c346d1b02863e, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:01:02.565 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,d79c346d1b02863e,3092c2ff77d65b45] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=d79c346d1b02863e, duration=16 ms] 2025-10-27 11:01:02.707 INFO 2458 --- [prometheus-minute-schedule-1,415358f9893517ef,8b3ca682d6086a4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:02.716 INFO 2458 --- [prometheus-minute-schedule-1,415358f9893517ef,8b3ca682d6086a4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:02.859 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:02.860 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:03.070 INFO 2458 --- [ocp-schedules-5,4eab6c76aac433b3,f8f2a8540921b035] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:03.073 INFO 2458 --- [ocp-schedules-5,4eab6c76aac433b3,f8f2a8540921b035] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:03.156 INFO 2458 --- [ocp-schedules-7,9a022ba50a3136e1,444391e65c7a3d56] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:03.252 WARN 2458 --- [scanner-112115233-1,9a022ba50a3136e1,ceb8ec1c8f7d05c8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:03.644 INFO 2458 --- [alarm-detect-10,9a022ba50a3136e1,175956760ea701c5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534030 ~ 1761534053 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534045, value=66.404)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534045, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534045=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:01:03.818 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,570f5050fe2aed55,e03b0c17fcd80f56] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=570f5050fe2aed55, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:01:03.819 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,570f5050fe2aed55,e03b0c17fcd80f56] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=570f5050fe2aed55, duration=1 ms] 2025-10-27 11:01:04.454 WARN 2458 --- [ocp-async-3,7a9299655fd311ea,44ca00b9ab2141ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:04.454 INFO 2458 --- [ocp-async-8,6855d411508dc83b,7094e3bf1c6934e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:05.537 INFO 2458 --- [ocp-schedules-3,e454b681da86c1fe,d6d7a006e470d51c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:05.707 INFO 2458 --- [prometheus-minute-schedule-1,46cb722233597250,6652c98bf47a9b72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:05.713 INFO 2458 --- [prometheus-minute-schedule-1,46cb722233597250,6652c98bf47a9b72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:05.860 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:05.861 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:06.454 WARN 2458 --- [ocp-async-6,ce3903706ad988f4,7a3cdf37e17bf945] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:06.454 INFO 2458 --- [ocp-async-7,fed326a48a0a0185,ea1d4f54990ceb57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:06.473 INFO 2458 --- [ocp-schedules-23,38dd7e686d5de72a,8d7ee6326d79bfa8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:01:06.481 INFO 2458 --- [ocp-schedules-23,38dd7e686d5de72a,8d7ee6326d79bfa8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:06.481 INFO 2458 --- [ocp-schedules-23,38dd7e686d5de72a,8d7ee6326d79bfa8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:07.405 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:01:07.405+08:00, tenantSize=6 2025-10-27 11:01:07.448 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761534067405000 2025-10-27 11:01:07.448 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761533767346000 2025-10-27 11:01:07.534 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:01:07.546 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761534067539000 2025-10-27 11:01:07.546 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761533767273000 2025-10-27 11:01:07.570 INFO 2458 --- [ocp-schedules-13,90f58167670d9571,4b08e3f8bce738b5] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:01:08.454 INFO 2458 --- [ocp-async-7,cc4ea96e7209dcbe,d3f06c170dcd3615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:08.454 WARN 2458 --- [ocp-async-1,774cae218716fc84,4dfcf5ced2d536b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:08.504 INFO 2458 --- [ocp-async-3,36cb36ea8a4a6ef4,709ebe3d95afce06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:08.707 INFO 2458 --- [prometheus-minute-schedule-1,cfc8b4387c87800c,0b09d49f1afd6b51] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:08.715 INFO 2458 --- [prometheus-minute-schedule-1,cfc8b4387c87800c,0b09d49f1afd6b51] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:08.861 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:08.862 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:10.013 INFO 2458 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:01:10.013 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:01:10.053 INFO 2458 --- [Check obproxy1,a6adf99c8c87acfd,0fe02a96acb82999] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:01:10.053 INFO 2458 --- [Check obproxy1,a6adf99c8c87acfd,0fe02a96acb82999] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:01:10.059 INFO 2458 --- [Collect obproxy1,0da3e5c8dba27956,ec8dd7392b9c6157] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:01:10.059 INFO 2458 --- [Collect obproxy1,0da3e5c8dba27956,ec8dd7392b9c6157] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:01:10.088 INFO 2458 --- [Check obproxy1,a6adf99c8c87acfd,0fe02a96acb82999] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:01:10.091 INFO 2458 --- [Check obproxy1,a6adf99c8c87acfd,0fe02a96acb82999] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:01:10.091 INFO 2458 --- [Check obproxy1,a6adf99c8c87acfd,0fe02a96acb82999] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:01:10.092 INFO 2458 --- [Collect obproxy1,0da3e5c8dba27956,ec8dd7392b9c6157] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:01:10.095 INFO 2458 --- [Collect obproxy1,0da3e5c8dba27956,ec8dd7392b9c6157] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:01:10.098 INFO 2458 --- [Collect obproxy1,0da3e5c8dba27956,ec8dd7392b9c6157] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:01:10.175 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:01:10.455 INFO 2458 --- [ocp-async-2,b34dcafe271713a7,a85f53cffc8b89e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:10.455 WARN 2458 --- [ocp-async-9,c439541ad24158ad,6259496eb67f62f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:11.707 INFO 2458 --- [prometheus-minute-schedule-1,ff873f59a23ceee8,d22e48e0ddc30978] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:11.715 INFO 2458 --- [prometheus-minute-schedule-1,ff873f59a23ceee8,d22e48e0ddc30978] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:11.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:11.717198+08:00, lastProcessTime=2025-10-27T11:01:01.716491+08:00 2025-10-27 11:01:11.862 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:11.863 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:11.960 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:11.717198+08:00 2025-10-27 11:01:12.455 INFO 2458 --- [ocp-async-4,7714095a03fd3cbf,03e9a0b2b4ffed21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:12.455 WARN 2458 --- [ocp-async-8,766d9ec4a6ae402f,35ba9a533099e28e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:12.761 INFO 2458 --- [ocp-schedules-13,edf1f01387036ba5,0e68b378aa5a6716] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:01:12.761 INFO 2458 --- [ocp-schedules-13,edf1f01387036ba5,0e68b378aa5a6716] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:01:12.769 INFO 2458 --- [ocp-schedules-13,edf1f01387036ba5,0e68b378aa5a6716] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:01:12.769 INFO 2458 --- [ocp-schedules-13,edf1f01387036ba5,0e68b378aa5a6716] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:01:13.081 INFO 2458 --- [ocp-schedules-25,c66dcc4202d0a395,b522e2927e22fa2f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:13.084 INFO 2458 --- [ocp-schedules-25,c66dcc4202d0a395,b522e2927e22fa2f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:13.431 INFO 2458 --- [ocp-schedules-18,16900bb61f8f7eb6,0d63ba19d52e627c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:01:13.431 INFO 2458 --- [ocp-schedules-18,16900bb61f8f7eb6,0d63ba19d52e627c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:01:13.444 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534000, nodeCount=821, affectRows=821 2025-10-27 11:01:13.490 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:01:13.491 INFO 2458 --- [Collect all tenants major compactions5,16900bb61f8f7eb6,9d47db4efbfce963] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:01:13.491 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:01:13.491 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:01:13.491 INFO 2458 --- [Collect all tenants major compactions6,16900bb61f8f7eb6,f1776b205387f0be] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:01:13.493 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:01:13.529 INFO 2458 --- [Collect all tenants major compactions6,16900bb61f8f7eb6,f1776b205387f0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.530 INFO 2458 --- [Collect all tenants major compactions5,16900bb61f8f7eb6,9d47db4efbfce963] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.531 INFO 2458 --- [Collect all tenants major compactions5,16900bb61f8f7eb6,9d47db4efbfce963] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:01:13.531 INFO 2458 --- [Collect all tenants major compactions6,16900bb61f8f7eb6,f1776b205387f0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:01:13.535 INFO 2458 --- [Collect all tenants major compactions5,16900bb61f8f7eb6,9d47db4efbfce963] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:01:13.535 INFO 2458 --- [Collect all tenants major compactions6,16900bb61f8f7eb6,f1776b205387f0be] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:01:13.550 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.550 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.550 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.550 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.550 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.551 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.552 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:13.553 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:13.554 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:01:13.556 INFO 2458 --- [Collect all tenants major compactions2,16900bb61f8f7eb6,8ef470c5bf47b14c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:01:13.557 INFO 2458 --- [Collect all tenants major compactions1,16900bb61f8f7eb6,a3bc9ef0dff2c18b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:01:13.558 INFO 2458 --- [Collect all tenants major compactions4,16900bb61f8f7eb6,e59b3bad525af79a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:01:13.558 INFO 2458 --- [Collect all tenants major compactions3,16900bb61f8f7eb6,7faef82799453b52] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:01:13.559 INFO 2458 --- [ocp-schedules-18,16900bb61f8f7eb6,0d63ba19d52e627c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:01:13.650 INFO 2458 --- [ocp-schedules-21,f3cf36af5ec30742,3102b15bbae47020] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:13.688 WARN 2458 --- [scanner-112117060-1,f3cf36af5ec30742,323faa74b595ba6a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:13.771 INFO 2458 --- [alarm-detect-20,f3cf36af5ec30742,8f8c5e3fc8a97463] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534045 ~ 1761534063 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534060, value=60.803)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534060, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534060=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:01:13.898 INFO 2458 --- [ocp-schedules-14,5c40b17f294cbc61,d1e9194f7bdd276b] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:01:13.898+08:00, clusters=[1, 2] 2025-10-27 11:01:13.901 INFO 2458 --- [ocp-schedules-14,5c40b17f294cbc61,d1e9194f7bdd276b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:01:13.902 INFO 2458 --- [ocp-schedules-14,5c40b17f294cbc61,d1e9194f7bdd276b] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:01:14.181 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=1000, affectRows=1000 2025-10-27 11:01:14.185 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=994, validCount=992 2025-10-27 11:01:14.236 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=1000, affectRows=1000 2025-10-27 11:01:14.240 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=999, validCount=995 2025-10-27 11:01:14.253 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587936) Duplicate entry '835370602992181248-1761533940' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587936) Duplicate entry '835370602992181248-1761533940' for key 'PRIMARY' 2025-10-27 11:01:14.455 INFO 2458 --- [ocp-async-5,e86eda1b5e45a8ca,5aa89bab5df01123] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:14.455 WARN 2458 --- [ocp-async-10,31b3417f759dfd4b,89d0b248fd35126b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:14.517 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=1000, affectRows=1000 2025-10-27 11:01:14.521 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=1000, validCount=996 2025-10-27 11:01:14.553 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=75, affectRows=75 2025-10-27 11:01:14.553 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=74, validCount=74 2025-10-27 11:01:14.577 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=1000, affectRows=1000 2025-10-27 11:01:14.581 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=996, validCount=992 2025-10-27 11:01:14.707 INFO 2458 --- [prometheus-minute-schedule-1,6637ce85515fdf9f,ee4d6ff5d4e7fcc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:14.714 INFO 2458 --- [prometheus-minute-schedule-1,6637ce85515fdf9f,ee4d6ff5d4e7fcc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:14.863 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:14.864 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:15.545 INFO 2458 --- [ocp-schedules-28,b6b244dfde98d142,80e1965190702c0b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:16.455 WARN 2458 --- [ocp-async-9,e8c4c67093c9dd17,a934e64554d7f425] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:16.455 INFO 2458 --- [ocp-async-3,3b412046648caec6,c05bfee1c8baea35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:17.271 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.277 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicSt 2025-10-27 11:01:17.330 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.332 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.339 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.346 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.352 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.355 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.358 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.361 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.365 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.371 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.375 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.380 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.381 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.382 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.383 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.384 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.385 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.387 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.388 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.389 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.390 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.391 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.392 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.394 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:55:17.129, end=2025-10-27T11:00:17.266} 2025-10-27 11:01:17.394 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:01:17.707 INFO 2458 --- [prometheus-minute-schedule-1,1d8d6d523b6e4f6e,6f753d1090d08b02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:17.718 INFO 2458 --- [prometheus-minute-schedule-1,1d8d6d523b6e4f6e,6f753d1090d08b02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:17.864 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:17.865 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:18.292 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534000, nodeCount=1000, affectRows=1000 2025-10-27 11:01:18.297 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534000, nodeCount=995, validCount=982 2025-10-27 11:01:18.455 WARN 2458 --- [ocp-async-6,ea917bfb5aca9d1a,5413c03f44e6281c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:18.455 INFO 2458 --- [ocp-async-8,e039dfb1e1fe0dce,5f3640a8581a5c8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:18.504 INFO 2458 --- [ocp-async-10,f2a99a8580ace567,c44bc9846010569e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:18.801 INFO 2458 --- [ocp-schedules-6,243406d259c35b1a,80d3d5b1c3d70c44] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:01:20.455 WARN 2458 --- [ocp-async-10,b406b49415eea0bf,0254de511c0c26cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:20.455 INFO 2458 --- [ocp-async-4,09aa8ca1f910deed,b2f8a0df93715f9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:20.707 INFO 2458 --- [prometheus-minute-schedule-1,2a229d2a1964d05c,6ab8a9dc2480c5ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:20.716 INFO 2458 --- [prometheus-minute-schedule-1,2a229d2a1964d05c,6ab8a9dc2480c5ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:20.865 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:20.866 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:21.411 INFO 2458 --- [ocp-schedules-25,876b297108ead39f,7b047a4d88b0b944] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:21.411 INFO 2458 --- [ocp-schedules-15,7898bee5b12fbe38,3307d87e5c0ac1cc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:21.411 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:01:21.411 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:01:21.411 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:01:21.411 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:01:21.412 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:01:21.425 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:21.426 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:21.426 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:21.426 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:21.429 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:21.430 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:21.430 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:21.430 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:21.431 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:21.431 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:21.431 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:21.431 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:21.431 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:21.432 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:21.432 INFO 2458 --- [ocp-schedules-30,e6beec453d4e88df,e8531d3e7be2036b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:21.433 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:21.433 INFO 2458 --- [host-check-clockdiff1,e6beec453d4e88df,b94ef0ba511771f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-clockdiff3,e6beec453d4e88df,1e09aea1642facb8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-clockdiff4,e6beec453d4e88df,1c5a83823c9ceaa4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-clockdiff2,e6beec453d4e88df,cc57754696b0945c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:01:21.434 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:21.435 INFO 2458 --- [host-check-clockdiff5,e6beec453d4e88df,8cd66e6c73a3b995] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:01:21.436 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:21.437 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:21.443 INFO 2458 --- [ocp-schedules-29,402f6e44bc0105c0,7af1115aadb37f33] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:01:21.453 INFO 2458 --- [host-check-clockdiff1,e6beec453d4e88df,b94ef0ba511771f9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534081 0 0 , error= 2025-10-27 11:01:21.459 INFO 2458 --- [host-check-clockdiff5,e6beec453d4e88df,8cd66e6c73a3b995] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534081 0 0 , error= 2025-10-27 11:01:21.466 INFO 2458 --- [host-check-clockdiff2,e6beec453d4e88df,cc57754696b0945c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534081 -1 -1 , error= 2025-10-27 11:01:21.468 INFO 2458 --- [host-check-clockdiff3,e6beec453d4e88df,1e09aea1642facb8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534081 0 0 , error= 2025-10-27 11:01:21.540 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:01:21.543 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:01:21.544 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:01:21.544 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:01:21.544 INFO 2458 --- [host-check-agent-status3,876b297108ead39f,49bacce803fdbe25] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:01:21.545 INFO 2458 --- [ocp-schedules-8,933a2bd4188a7f10,e750a632eed3649b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:01:21.547 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:01:21.547 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:01:21.547 INFO 2458 --- [host-check-agent-status4,876b297108ead39f,36ff94ff4887705a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:01:21.549 INFO 2458 --- [ocp-schedules-12,183e0a4fb5f3a502,f86147bf74015108] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:01:21.550 INFO 2458 --- [ocp-schedules-8,933a2bd4188a7f10,e750a632eed3649b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:21.553 INFO 2458 --- [ocp-schedules-12,183e0a4fb5f3a502,f86147bf74015108] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:21.553 INFO 2458 --- [ocp-schedules-23,7e8b390adae76d8f,a09ab537af128e1d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:01:21.554 INFO 2458 --- [ocp-schedules-12,183e0a4fb5f3a502,f86147bf74015108] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:01:21.558 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:01:21.558 INFO 2458 --- [ocp-schedules-23,7e8b390adae76d8f,a09ab537af128e1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:21.559 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:01:21.560 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:01:21.560112809+08:00 2025-10-27 11:01:21.561 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:01:21.560+08:00 2025-10-27 11:01:21.562 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:01:21.561351742+08:00 2025-10-27 11:01:21.565 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:01:21.562+08:00 2025-10-27 11:01:21.566 INFO 2458 --- [ocp-schedules-17,b1e3faecc8af25bf,42b41cdb01b1b90e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:01:21.568 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:01:21.571 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:01:21.571 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:01:21.571 INFO 2458 --- [host-check-agent-status1,876b297108ead39f,f39e3b79e10eac2e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:01:21.572 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:01:21.560+08:00 and status AVAILABLE 2025-10-27 11:01:21.578 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:01:21.582 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:01:21.562+08:00 and status AVAILABLE 2025-10-27 11:01:21.587 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:01:21.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:21.599 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:01:21.616 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:21.618 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:01:21.619 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:21.619 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:01:21.618438248+08:00 2025-10-27 11:01:21.619 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:01:21.619+08:00 2025-10-27 11:01:21.619 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:01:21.622 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:01:21.622 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:01:21.622 INFO 2458 --- [host-check-agent-status2,876b297108ead39f,96fb29e9ae899276] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:01:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.627 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:21.628 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:01:21.629 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:01:21.619+08:00 and status AVAILABLE 2025-10-27 11:01:21.630 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:01:21.630649339+08:00 2025-10-27 11:01:21.630 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:21.631 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:01:21.630+08:00 2025-10-27 11:01:21.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:01:21.635 INFO 2458 --- [ocp-schedules-8,933a2bd4188a7f10,e750a632eed3649b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:01:21.637 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:01:21.641 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:01:21.630+08:00 and status AVAILABLE 2025-10-27 11:01:21.644 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:01:21.647 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:01:21.647 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:01:21.647 INFO 2458 --- [host-check-agent-status5,876b297108ead39f,3c2946e193f91a1d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:01:21.648 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:01:21.651 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:01:21.652 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:01:21.651506295+08:00 2025-10-27 11:01:21.652 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:01:21.652+08:00 2025-10-27 11:01:21.653 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.654 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:21.665 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:01:21.652+08:00 and status AVAILABLE 2025-10-27 11:01:21.670 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:21.671 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:01:21.673 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:21.676 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:21.679 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:21.685 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.685 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:21.690 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:21.695 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:21.704 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:21.707 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:21.711 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:01:21.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:21.718133+08:00, lastProcessTime=2025-10-27T11:01:11.717198+08:00 2025-10-27 11:01:21.729 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:01:21.742 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:01:21.747 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:21.748 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:01:21.754 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:21.755 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:21.765 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:01:21.768 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:21.769 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:01:21.773 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:01:21.790 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:01:21.791 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:21.792 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:01:21.807 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:21.808 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:01:21.810 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:01:21.818 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:21.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:21.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:21.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.826 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:01:21.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:21.830 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:01:21.839 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:01:21.850 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:21.857 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:01:21.860 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:21.863 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:01:21.866 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:01:21.874 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:21.877 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:01:21.888 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:01:21.889 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:21.890 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:21.893 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.895 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:01:21.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:01:21.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:21.902 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:01:21.907 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:21.908 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:21.913 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:01:21.928 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:01:21.928 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:21.929 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:01:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:21.936 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:21.937 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:21.938 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:21.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:01:21.947 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:21.954 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:01:21.957 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:01:21.959 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:21.961 INFO 2458 --- [host-check-host-status2,7898bee5b12fbe38,dc55beef59d9841c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:01:21.963 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:01:21.964 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:21.967 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:21.973 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:21.976 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:01:21.981 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:21.981 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:21.992 INFO 2458 --- [host-check-host-status4,7898bee5b12fbe38,5ac30a09dd2d364c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:01:21.994 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:01:21.994 INFO 2458 --- [host-check-host-status1,7898bee5b12fbe38,9d76f616c1127cc5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:01:22.009 INFO 2458 --- [host-check-host-status5,7898bee5b12fbe38,83a7584dd12ada3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:01:22.014 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:22.014 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:22.021 INFO 2458 --- [host-check-host-status3,7898bee5b12fbe38,409ec3c119c6ff0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:01:22.027 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:01:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:22.069 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:01:22.070 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.071 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:01:22.072 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.073 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:01:22.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:22.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:22.089 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:01:22.113 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:22.114 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.114 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:01:22.122 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:22.147 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:01:22.147 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761533484668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:01:22.168 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:22.169 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.169 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:22.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:01:22.190 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:01:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:22.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:22.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:01:22.324 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:01:22.325 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.325 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:01:22.327 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.328 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:01:22.349 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:21.718133+08:00 2025-10-27 11:01:22.360 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:01:22.409 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:22.410 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:22.411 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:01:22.447 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:01:22.447 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:01:22.447 INFO 2458 --- [ocp-schedules-23,7e8b390adae76d8f,a09ab537af128e1d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:01:22.455 WARN 2458 --- [ocp-async-4,e9e1133004b4b010,06c9c1b41fafb082] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:22.455 INFO 2458 --- [ocp-async-5,25994eea84b640d8,4476fd27e80cfe50] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:23.099 INFO 2458 --- [ocp-schedules-17,2737d144b5a6513a,99b2898039eeb7ca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:23.102 INFO 2458 --- [ocp-schedules-17,2737d144b5a6513a,99b2898039eeb7ca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:23.231 INFO 2458 --- [host-check-clockdiff4,e6beec453d4e88df,1c5a83823c9ceaa4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534083 0 0 , error= 2025-10-27 11:01:23.707 INFO 2458 --- [prometheus-minute-schedule-1,83ef54b6606d4b32,e8bda59ce4b984c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:23.716 INFO 2458 --- [prometheus-minute-schedule-1,83ef54b6606d4b32,e8bda59ce4b984c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:23.777 INFO 2458 --- [ocp-schedules-14,b36aa19f4b80138a,ffc57414448e0ca1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:23.835 WARN 2458 --- [scanner-112117237-1,b36aa19f4b80138a,7a673954e7eeec78] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:23.866 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:23.867 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:24.455 WARN 2458 --- [ocp-async-6,1913b498f41e630c,f84682de32f503dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:24.456 INFO 2458 --- [ocp-async-2,485754f41a30645d,23ca4d7800cec61c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:24.667 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,30d60d514a36f912,5bc2f369e4901c36] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&version=2&ObClusterId=1760510188, client=10.40.66.107, traceId=30d60d514a36f912, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase2,1760510188,2,{"ObRegion":"myoceanbase2","ObCluster":"myoceanbase2","ObRegionId":1760510188,"ObClusterId":1760510188,"Type":"PRIMARY","timestamp":1761534084666393,"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:01:24.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,30d60d514a36f912,5bc2f369e4901c36] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase2, obClusterId=1760510188, rootServerJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, groupName=myoceanbase2, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:01:24.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,30d60d514a36f912,5bc2f369e4901c36] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:01:24.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,30d60d514a36f912,5bc2f369e4901c36] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase2, clusterId=1760510188, affectRows=1 2025-10-27 11:01:24.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,30d60d514a36f912,5bc2f369e4901c36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.66.107, traceId=30d60d514a36f912, duration=9 ms] 2025-10-27 11:01:25.551 INFO 2458 --- [ocp-schedules-26,24f8b19a0cee2f7d,28935cf6ccc28940] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:26.456 WARN 2458 --- [ocp-async-10,468cbeaae5ec351d,692221b2fdba46ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:26.456 INFO 2458 --- [ocp-async-8,dc194af018c46712,df3cde84d7af6747] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:26.484 INFO 2458 --- [ocp-schedules-11,6e535c92a866b8e9,27116da852211f89] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:01:26.491 INFO 2458 --- [ocp-schedules-11,6e535c92a866b8e9,27116da852211f89] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:26.491 INFO 2458 --- [ocp-schedules-11,6e535c92a866b8e9,27116da852211f89] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:26.707 INFO 2458 --- [prometheus-minute-schedule-1,172978a57c85825c,581447a4e9530bfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:26.716 INFO 2458 --- [prometheus-minute-schedule-1,172978a57c85825c,581447a4e9530bfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:26.867 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:26.868 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:28.456 INFO 2458 --- [ocp-async-9,8686c8ee9a4280dc,55aa920ff4a9bd8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:28.456 WARN 2458 --- [ocp-async-4,17a6c082ef2488b6,b5e5d3ba005256f0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:28.504 INFO 2458 --- [ocp-async-1,bd9dc2101e1ea260,a0a33da89a7c0715] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:28.774 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:01:28.774 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:01:28.776 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:01:28.776 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:01:28.776 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:01:28.776 INFO 2458 --- [ocp-schedules-30,4224f93c16437882,6658ac73eef10b20] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:01:29.265 INFO 2458 --- [ocp-schedules-18,f03c27aad2ea3762,a5297b68960b161c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:01:29.275 INFO 2458 --- [ocp-schedules-18,f03c27aad2ea3762,a5297b68960b161c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:01:29.276 INFO 2458 --- [ocp-schedules-18,f03c27aad2ea3762,a5297b68960b161c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:01:29.707 INFO 2458 --- [prometheus-minute-schedule-1,bc647b2729204493,7e5e48f87c435351] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:29.718 INFO 2458 --- [prometheus-minute-schedule-1,bc647b2729204493,7e5e48f87c435351] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:29.868 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:29.869 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:30.456 WARN 2458 --- [ocp-async-6,04c5f811e7f4d87f,b6e0dfb81618d177] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:30.456 INFO 2458 --- [ocp-async-1,447b804f0276458a,8217b6d315f2e3fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:31.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:31.721234+08:00, lastProcessTime=2025-10-27T11:01:21.718133+08:00 2025-10-27 11:01:32.235 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:31.721234+08:00 2025-10-27 11:01:32.456 INFO 2458 --- [ocp-async-10,3414f80d96bdc94d,8b3719321660feec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:32.456 WARN 2458 --- [ocp-async-7,83c3ccabae781ff1,00c2a06148c88d2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:32.707 INFO 2458 --- [prometheus-minute-schedule-1,8c0329384e824a49,3b4c27ff7e9fb1f2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:32.718 INFO 2458 --- [prometheus-minute-schedule-1,8c0329384e824a49,3b4c27ff7e9fb1f2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:32.869 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:32.870 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:33.110 INFO 2458 --- [ocp-schedules-7,3066eaf51ca0d08f,34c4ad024b043a7c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:33.114 INFO 2458 --- [ocp-schedules-7,3066eaf51ca0d08f,34c4ad024b043a7c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:33.680 INFO 2458 --- [ocp-schedules-11,2b1501e94c932e4d,9375a0ddfc57639a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:01:33.680 INFO 2458 --- [ocp-schedules-11,2b1501e94c932e4d,9375a0ddfc57639a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:01:33.735 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:01:33.735 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:01:33.846 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:33.847 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:01:33.860 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:33.861 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:01:33.930 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:33.931 INFO 2458 --- [ocp-schedules-27,15413393f231306a,b5ad8299b264594d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:01:33.948 INFO 2458 --- [ocp-schedules-6,ef9421df03850ffe,72197a250ad96713] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:33.967 WARN 2458 --- [scanner-112117485-1,ef9421df03850ffe,f961eb6f2a2601dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:34.067 INFO 2458 --- [alarm-detect-1,ef9421df03850ffe,d71c341b7f25d399] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534060 ~ 1761534084 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534075, value=46.894)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534075, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534075=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:01:34.456 INFO 2458 --- [ocp-async-3,9ece9ce2f799b226,51f052cc2d31845a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:34.456 WARN 2458 --- [ocp-async-9,3f6fb6bec1019e52,15e6251053ef5e04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:35.560 INFO 2458 --- [ocp-schedules-26,e16cdfab645a52bf,ac5aca66b494af56] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:35.707 INFO 2458 --- [prometheus-minute-schedule-1,a63254cee26c37b6,02a91e00a7db9f97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:35.717 INFO 2458 --- [prometheus-minute-schedule-1,a63254cee26c37b6,02a91e00a7db9f97] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:35.805 INFO 2458 --- [ocp-schedules-6,ae5faa0baaba2141,1c8219b5f6224a4f] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-10-27 11:01:35.813 INFO 2458 --- [ocp-schedules-6,ae5faa0baaba2141,1c8219b5f6224a4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:35.870 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:35.870 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:35.884 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:35.888 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 02:04:47.657729, 2025-10-27 11:01:35.839, 1000] 2025-10-27 11:01:35.889 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2025-10-27 02:04:47.657729, endTime 2025-10-27 11:01:35.839 2025-10-27 11:01:35.901 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:35.904 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091391, 2025-10-27 11:01:35.844, 1000] 2025-10-27 11:01:35.906 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2025-10-27 08:00:44.091391, endTime 2025-10-27 11:01:35.844 2025-10-27 11:01:35.932 INFO 2458 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:35.936 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:35.958 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-17 15:45:19.719943, 2025-10-27 11:01:35.909, 1000] 2025-10-27 11:01:35.959 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2025-10-17 15:45:19.719943, endTime 2025-10-27 11:01:35.909 2025-10-27 11:01:35.984 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:35.984 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:36.007 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091611, 2025-10-27 11:01:35.939, 1000] 2025-10-27 11:01:36.008 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2025-10-27 08:00:44.091611, endTime 2025-10-27 11:01:35.939 2025-10-27 11:01:36.008 INFO 2458 --- [ocp-schedules-6,ae5faa0baaba2141,1c8219b5f6224a4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=2, successCount=2 2025-10-27 11:01:36.338 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:36.338 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:01:36.338 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:01:36.339 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:01:36.339 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:01:36.339 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:01:36.353 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:36.353 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:36.354 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:36.354 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:36.354 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:36.357 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:36.357 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:36.359 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:36.359 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:36.359 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:36.414 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:01:36.414 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:01:36.415 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:01:36.417 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:01:36.420 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:01:36.422 INFO 2458 --- [ocp-schedules-30,0208840f9a233a34,0773ede4ece22f21] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:01:36.456 WARN 2458 --- [ocp-async-2,5285cfc4ba082a5d,b87657ba982e75e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:36.456 INFO 2458 --- [ocp-async-5,3be4cbefe348ccaf,1d68ed751b3363aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:37.708 INFO 2458 --- [prometheus-cache-monitor1,62dbbb5a5a475ad8,a82231a60862458a] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20287,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5074,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":15713} 2025-10-27 11:01:37.708 INFO 2458 --- [prometheus-cache-monitor1,62dbbb5a5a475ad8,a82231a60862458a] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:01:38.456 INFO 2458 --- [ocp-async-7,30ce7763524c8d81,b9cd0d92321b7a25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:38.456 WARN 2458 --- [ocp-async-8,ce07b98ed84fc76c,008b49fec61b0adf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:38.504 INFO 2458 --- [ocp-async-10,e84e7f45fb847595,1de15dd0af488841] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:38.544 INFO 2458 --- [ocp-schedules-13,68962318aceb5377,969a349ebef52b62] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:01:38.550 INFO 2458 --- [ocp-schedules-13,68962318aceb5377,969a349ebef52b62] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:38.550 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:01:38.550 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:01:38.550 INFO 2458 --- [ocp-schedules-13,68962318aceb5377,969a349ebef52b62] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:01:38.707 INFO 2458 --- [prometheus-minute-schedule-1,431bbaec2b85d09f,7a913b59c1481a7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:38.714 INFO 2458 --- [prometheus-minute-schedule-1,431bbaec2b85d09f,7a913b59c1481a7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:38.870 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:38.871 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:40.009 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:01:40.009 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:01:40.044 INFO 2458 --- [Check obproxy1,519dbbad530b55f5,a94b53e82da01e23] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:01:40.044 INFO 2458 --- [Check obproxy1,519dbbad530b55f5,a94b53e82da01e23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:01:40.044 INFO 2458 --- [Collect obproxy1,e093242991cda358,cb5664f751b03db7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:01:40.044 INFO 2458 --- [Collect obproxy1,e093242991cda358,cb5664f751b03db7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:01:40.079 INFO 2458 --- [Collect obproxy1,e093242991cda358,cb5664f751b03db7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:01:40.080 INFO 2458 --- [Check obproxy1,519dbbad530b55f5,a94b53e82da01e23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:01:40.083 INFO 2458 --- [Collect obproxy1,e093242991cda358,cb5664f751b03db7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:01:40.084 INFO 2458 --- [Check obproxy1,519dbbad530b55f5,a94b53e82da01e23] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:01:40.084 INFO 2458 --- [Check obproxy1,519dbbad530b55f5,a94b53e82da01e23] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:01:40.085 INFO 2458 --- [Collect obproxy1,e093242991cda358,cb5664f751b03db7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:01:40.099 INFO 2458 --- [ocp-schedules-30,4f5e5119dcca26b2,803d45cc9c892f06] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:01:40.100 INFO 2458 --- [ocp-schedules-30,4f5e5119dcca26b2,803d45cc9c892f06] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:01:40.457 WARN 2458 --- [ocp-async-4,2650624eb7eb69e8,5001defed028a8c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:40.457 INFO 2458 --- [ocp-async-3,f665b32cab2eced2,84da398e5e50744d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:41.707 INFO 2458 --- [prometheus-minute-schedule-1,f9afd2bfdab47897,ed58ab16d7ffbd95] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:41.714 INFO 2458 --- [prometheus-minute-schedule-1,f9afd2bfdab47897,ed58ab16d7ffbd95] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:41.726 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:41.725145+08:00, lastProcessTime=2025-10-27T11:01:31.721234+08:00 2025-10-27 11:01:41.871 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:41.872 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:41.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:41.725145+08:00 2025-10-27 11:01:42.457 WARN 2458 --- [ocp-async-6,9d50a80ecbe3f922,c2597a89fe4429e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:42.457 INFO 2458 --- [ocp-async-5,155e855dcbc1ecbe,25f76789142e644f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:42.912 INFO 2458 --- [ocp-schedules-17,404cb03164cbb40d,1684b3ce26b02cfe] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:01:42.912 INFO 2458 --- [ocp-schedules-17,404cb03164cbb40d,1684b3ce26b02cfe] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:01:42.919 INFO 2458 --- [ocp-schedules-17,404cb03164cbb40d,1684b3ce26b02cfe] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:01:42.919 INFO 2458 --- [ocp-schedules-17,404cb03164cbb40d,1684b3ce26b02cfe] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:01:43.122 INFO 2458 --- [ocp-schedules-25,acc5c93b6ba10c2b,702863f5d5bbc258] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:43.125 INFO 2458 --- [ocp-schedules-25,acc5c93b6ba10c2b,702863f5d5bbc258] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:43.921 INFO 2458 --- [ocp-schedules-29,e6efa83ed4a18904,d0762858d221c8de] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:01:44.075 INFO 2458 --- [ocp-schedules-28,3d92e81fac02b409,9d1bd7d88863f97c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:44.103 WARN 2458 --- [scanner-112117663-1,3d92e81fac02b409,bf2e6f0b46d1d58f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:44.258 INFO 2458 --- [alarm-detect-3,3d92e81fac02b409,43cadc51a8fefaf5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534075 ~ 1761534094 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534090, value=49.537)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534090, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534090=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:01:44.457 WARN 2458 --- [ocp-async-1,30301e83b2d1a736,820e07a6fe9b9d80] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:44.457 INFO 2458 --- [ocp-async-7,6f3f8c61e04f9ff0,578d1dfdb32fc0ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:44.707 INFO 2458 --- [prometheus-minute-schedule-1,78feefdb1dbf2732,63da78b55530e9ad] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:44.715 INFO 2458 --- [prometheus-minute-schedule-1,78feefdb1dbf2732,63da78b55530e9ad] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:44.872 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:44.873 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:45.102 INFO 2458 --- [prometheus-second-schedule-1,7b11a5237a5c417b,904d65950b664f10] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:01:45.216 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,60c305d9ff28f9ff,19e63549ffffa49d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=60c305d9ff28f9ff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:01:45.216 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,60c305d9ff28f9ff,19e63549ffffa49d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=60c305d9ff28f9ff, duration=1 ms] 2025-10-27 11:01:45.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f54814451daa40bf,ebb9ea42598f88ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest, client=10.40.67.22, traceId=f54814451daa40bf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,false,] 2025-10-27 11:01:45.218 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f54814451daa40bf,ebb9ea42598f88ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=f54814451daa40bf, duration=1 ms] 2025-10-27 11:01:45.569 INFO 2458 --- [ocp-schedules-8,e18f838dcd9df751,c0ac45c971840c76] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:46.457 WARN 2458 --- [ocp-async-4,f6630ed92941ca77,35bc0d80b76ef4d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:46.457 INFO 2458 --- [ocp-async-10,07492dd53f6c7380,c1788e4f03eec0a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:46.494 INFO 2458 --- [ocp-schedules-5,6552b2d95b96ee4e,312c1b9190b062a0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:01:46.501 INFO 2458 --- [ocp-schedules-5,6552b2d95b96ee4e,312c1b9190b062a0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:46.501 INFO 2458 --- [ocp-schedules-5,6552b2d95b96ee4e,312c1b9190b062a0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:01:47.238 INFO 2458 --- [ocp-schedules-22,e106bb7979ec94c4,4006b7e4f3c98b21] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:01:47.238+08:00, clusters=[1, 2] 2025-10-27 11:01:47.254 INFO 2458 --- [ocp-schedules-22,e106bb7979ec94c4,4006b7e4f3c98b21] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:01:47.257 INFO 2458 --- [ocp-schedules-22,e106bb7979ec94c4,4006b7e4f3c98b21] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:01:47.707 INFO 2458 --- [prometheus-minute-schedule-1,fd38da8f6200bff1,95014840e267835d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:47.714 INFO 2458 --- [prometheus-minute-schedule-1,fd38da8f6200bff1,95014840e267835d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:47.873 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:47.873 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:48.457 INFO 2458 --- [ocp-async-6,92898d41f0790521,a47aa15dc5d89013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:48.457 WARN 2458 --- [ocp-async-2,d05e0055c6a07a8b,60da73c77eb41c92] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:48.504 INFO 2458 --- [ocp-async-5,049b47343fa0f598,a74f42f796fc0abb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:50.457 INFO 2458 --- [ocp-async-1,fbcee1918320f41c,d9f53bcfc7caa95a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:50.457 WARN 2458 --- [ocp-async-7,8d58fe4f84e95b95,15704a0b99819ffa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:50.707 INFO 2458 --- [prometheus-minute-schedule-1,bb90c8b15e2f7cff,b8613d7adf849b92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:50.714 INFO 2458 --- [prometheus-minute-schedule-1,bb90c8b15e2f7cff,b8613d7adf849b92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:50.873 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:50.874 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:51.410 INFO 2458 --- [ocp-schedules-10,4afa165db7ccea42,7c50068d26f0574d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:51.411 INFO 2458 --- [ocp-schedules-12,9431c515d1fd5957,7f3201699c46ec1a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:01:51.411 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:01:51.412 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:01:51.412 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:01:51.412 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:01:51.412 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:01:51.425 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:51.426 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:51.426 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:51.427 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:51.427 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:51.427 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:51.427 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:51.428 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:51.430 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:51.431 INFO 2458 --- [ocp-schedules-4,f030bc9e475920f4,da687fe6b0809df1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-clockdiff1,f030bc9e475920f4,d6ae413b8d025c68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:01:51.431 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:51.432 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:51.432 INFO 2458 --- [host-check-clockdiff2,f030bc9e475920f4,af5f78d004214f24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:01:51.432 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:51.432 INFO 2458 --- [host-check-clockdiff3,f030bc9e475920f4,50e004b8a076531f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:01:51.432 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:51.433 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:51.433 INFO 2458 --- [host-check-clockdiff4,f030bc9e475920f4,bd75a01ea4512e3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:01:51.434 INFO 2458 --- [host-check-clockdiff5,f030bc9e475920f4,f76fa6017517b804] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:01:51.434 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:51.443 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:51.448 INFO 2458 --- [ocp-schedules-27,4d21bb6fd103892a,ad2501d677ea418c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:01:51.446 INFO 2458 --- [host-check-clockdiff5,f030bc9e475920f4,f76fa6017517b804] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534111 0 0 , error= 2025-10-27 11:01:51.449 INFO 2458 --- [host-check-clockdiff3,f030bc9e475920f4,50e004b8a076531f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534111 -1 -1 , error= 2025-10-27 11:01:51.463 INFO 2458 --- [host-check-clockdiff1,f030bc9e475920f4,d6ae413b8d025c68] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534111 0 0 , error= 2025-10-27 11:01:51.465 INFO 2458 --- [host-check-clockdiff2,f030bc9e475920f4,af5f78d004214f24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534111 -1 -1 , error= 2025-10-27 11:01:51.514 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:01:51.515 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:01:51.516 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:01:51.516 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:01:51.517 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:01:51.517495550+08:00 2025-10-27 11:01:51.518 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:01:51.518 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:01:51.517+08:00 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status5,9431c515d1fd5957,3951acd8f741f54b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:01:51.519 INFO 2458 --- [host-check-agent-status3,9431c515d1fd5957,b6daac8c54fa7d98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:01:51.520 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:01:51.519990800+08:00 2025-10-27 11:01:51.521 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:01:51.520+08:00 2025-10-27 11:01:51.525 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:01:51.525 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:01:51.525 INFO 2458 --- [host-check-agent-status1,9431c515d1fd5957,470ed81bd162d34c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:01:51.527 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:01:51.517+08:00 and status AVAILABLE 2025-10-27 11:01:51.531 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:01:51.520+08:00 and status AVAILABLE 2025-10-27 11:01:51.532 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:01:51.535 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:01:51.536 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:01:51.537 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:01:51.537525338+08:00 2025-10-27 11:01:51.537 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:01:51.537 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:01:51.538 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:01:51.537754914+08:00 2025-10-27 11:01:51.538 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:01:51.537+08:00 2025-10-27 11:01:51.538 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:01:51.538+08:00 2025-10-27 11:01:51.539 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:01:51.538612934+08:00 2025-10-27 11:01:51.540 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:01:51.539+08:00 2025-10-27 11:01:51.543 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:01:51.545 INFO 2458 --- [ocp-schedules-29,0d08a3a438aff201,339d711925c30cda] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:01:51.546 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:01:51.546 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:01:51.546 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:01:51.546 INFO 2458 --- [host-check-agent-status2,9431c515d1fd5957,bb41b0ac97ed8a31] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:01:51.546 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:01:51.537+08:00 and status AVAILABLE 2025-10-27 11:01:51.547 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:01:51.538+08:00 and status AVAILABLE 2025-10-27 11:01:51.548 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:01:51.548 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:01:51.548 INFO 2458 --- [host-check-agent-status4,9431c515d1fd5957,d2c08ad2971b4cbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:01:51.549 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:01:51.539+08:00 and status AVAILABLE 2025-10-27 11:01:51.549 INFO 2458 --- [ocp-schedules-3,6e190b6b54a3b054,c13d0e325a1e0a72] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:01:51.551 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:01:51.553 INFO 2458 --- [ocp-schedules-3,6e190b6b54a3b054,c13d0e325a1e0a72] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:51.553 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:01:51.553 INFO 2458 --- [ocp-schedules-3,6e190b6b54a3b054,c13d0e325a1e0a72] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:01:51.554 INFO 2458 --- [ocp-schedules-5,72c587fd62d621d8,a1e8370f79b4e4d0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:01:51.555 INFO 2458 --- [ocp-schedules-29,0d08a3a438aff201,339d711925c30cda] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:51.555 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:01:51.555 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:01:51.557 INFO 2458 --- [ocp-schedules-5,72c587fd62d621d8,a1e8370f79b4e4d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:51.558 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:01:51.560 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:01:51.563 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:01:51.565 INFO 2458 --- [ocp-schedules-20,31d212d20bff6863,c8d01250e4f5e919] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:01:51.568 INFO 2458 --- [ocp-schedules-13,760a22067524deb7,63a69a992448ec8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:01:51.569 INFO 2458 --- [ocp-schedules-20,c64d947f3889a678,0e747902c67d41c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:01:51.575 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:01:51.576 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:51.578 INFO 2458 --- [ocp-schedules-13,760a22067524deb7,63a69a992448ec8f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:01:51.578 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:01:51.579 INFO 2458 --- [ocp-schedules-20,c64d947f3889a678,0e747902c67d41c3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:01:51.581 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:51.584 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:51.586 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:01:51.628 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.629 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:51.636 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:01:51.636 INFO 2458 --- [ocp-schedules-29,0d08a3a438aff201,339d711925c30cda] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:01:51.639 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:51.640 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:51.654 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:51.654 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:01:51.655 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:01:51.659 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:01:51.668 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:01:51.672 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:01:51.676 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:01:51.694 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.694 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.695 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.695 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:01:51.695 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:51.697 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:51.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.698 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:01:51.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:01:51.701 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:01:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:01:51.715929+08:00, lastProcessTime=2025-10-27T11:01:41.725145+08:00 2025-10-27 11:01:51.721 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:01:51.725 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:01:51.726 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:51.741 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:51.744 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:51.748 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:01:51.756 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:51.759 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:01:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:51.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:51.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.767 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:01:51.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:01:51.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:51.772 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:01:51.776 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:01:51.778 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:51.778 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:01:51.783 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:01:51.787 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:01:51.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:51.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:51.795 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:01:51.803 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:01:51.811 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:01:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:51.815 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:51.820 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:01:51.828 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:51.831 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:51.836 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:01:51.842 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:01:51.845 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:51.846 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:51.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:01:51.852 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:01:51.861 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:01:51.865 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:01:51.869 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:01:51.870 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:01:51.875 INFO 2458 --- [host-check-host-status3,4afa165db7ccea42,81e96c475d7e833e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:01:51.876 INFO 2458 --- [host-check-host-status4,4afa165db7ccea42,634f0ca6411e38e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:01:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:01:51.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:01:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.883 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:01:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:01:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:51.896 INFO 2458 --- [host-check-host-status1,4afa165db7ccea42,8f0375fbaf9f9e66] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:01:51.903 INFO 2458 --- [host-check-host-status2,4afa165db7ccea42,a0c9c98d991f501c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:01:51.908 INFO 2458 --- [host-check-host-status5,4afa165db7ccea42,4fd9d8c91208efe5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:01:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:51.950 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:01:51.969 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:51.969 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:51.975 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:01:51.715929+08:00 2025-10-27 11:01:51.994 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:01:51.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.995 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:01:51.997 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:51.998 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:01:52.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:52.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:52.015 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:01:52.038 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:52.039 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:52.040 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:01:52.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:52.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:01:52.066 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:01:52.066 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:01:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:01:52.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:01:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:52.160 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:01:52.163 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:01:52.175 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:01:52.185 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:01:52.188 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:01:52.235 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:01:52.285 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:01:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:01:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:52.289 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:01:52.322 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:52.364 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:01:52.380 INFO 2458 --- [host-check-clockdiff4,f030bc9e475920f4,bd75a01ea4512e3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534112 0 0 , error= 2025-10-27 11:01:52.405 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:01:52.405 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:01:52.405 INFO 2458 --- [ocp-schedules-5,72c587fd62d621d8,a1e8370f79b4e4d0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:01:52.458 INFO 2458 --- [ocp-async-2,ca5c215870750e2a,6c2b32ecee51009e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:52.458 WARN 2458 --- [ocp-async-3,df9f5a1791565995,4a9ea4644dadbe0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:53.133 INFO 2458 --- [ocp-schedules-20,af1e5ba887b0f4f0,f2baff709009b804] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:01:53.136 INFO 2458 --- [ocp-schedules-20,af1e5ba887b0f4f0,f2baff709009b804] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:01:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:01:53.219 INFO 2458 --- [ocp-schedules-26,748cfdc860e2a7f4,b71151e5574fc587] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20287,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5074,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":15713} 2025-10-27 11:01:53.219 INFO 2458 --- [ocp-schedules-26,748cfdc860e2a7f4,b71151e5574fc587] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:01:53.707 INFO 2458 --- [prometheus-minute-schedule-1,8266c2e602edfc89,ad94312613a7602f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:53.715 INFO 2458 --- [prometheus-minute-schedule-1,8266c2e602edfc89,ad94312613a7602f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:53.874 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:53.875 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:53.993 INFO 2458 --- [ocp-schedules-21,e3b1e1dab142c256,e055f8d9466822a2] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:01:54.001 INFO 2458 --- [ocp-schedules-21,e3b1e1dab142c256,e055f8d9466822a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:54.089 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:01:54.090 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.093 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:01:54.102 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.103 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.130 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.130 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.141 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.141 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.142 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:01:54.142 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.142 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.142 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.145 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:01:54.147 INFO 2458 --- [ocp-schedules-24,8a50995970290602,603b2ee67dcdd060] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:01:54.159 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.159 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.161 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.161 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.161 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.161 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.162 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.162 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.193 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.194 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.195 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.195 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.196 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.220 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.221 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.222 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.222 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.222 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.223 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:01:54.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.225 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.226 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.250 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.251 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.252 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.253 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.254 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.254 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.255 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.256 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.257 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.257 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.258 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.259 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.260 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:01:54.263 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:01:54.264 INFO 2458 --- [ocp-schedules-25,a1f6b6b38c12bb81,8fc01303684d4292] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:01:54.289 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.289 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.289 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.290 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:01:54.292 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.292 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.292 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.293 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:01:54.322 WARN 2458 --- [scanner-112117950-1,a1f6b6b38c12bb81,8a5f731e2ef94326] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:01:54.324 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:54.325 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.325 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:01:54.331 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.331 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.331 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.332 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:01:54.335 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.362 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.362 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.362 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.363 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:01:54.365 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.365 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.335 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:01:54.366 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.366 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:01:54.366 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.367 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:01:54.369 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:01:54.369 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:01:54.402 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.402 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.402 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.403 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:01:54.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.435 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.436 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:01:54.458 INFO 2458 --- [ocp-async-5,c3a0dc70fefe1144,a064e9d5e0c68bc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:54.458 WARN 2458 --- [ocp-async-7,e9f28551c86d1d42,90e9dc88f196bc22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:54.498 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.498 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.499 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.500 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.500 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.500 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.500 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.501 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.502 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.502 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.502 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.503 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.503 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.503 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.504 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.505 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.505 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.505 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.505 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.506 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.507 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.507 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.508 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.508 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.509 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.509 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.509 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.510 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.510 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.511 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.511 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.511 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.512 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.512 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.512 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.513 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.513 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:01:54.514 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.514 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.514 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.515 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:01:54.515 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.515 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.515 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.516 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:01:54.517 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:01:54.519 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:01:54.617 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:01:54.617 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.617 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.618 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:01:54.618 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:01:54.618 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:01:54.618 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:01:54.648 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:01:54.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:01:54.652 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:01:54.652 INFO 2458 --- [ocp-schedules-21,e3b1e1dab142c256,e055f8d9466822a2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:01:55.576 INFO 2458 --- [ocp-schedules-24,2a9a43500bfd73cd,d7da33b5e8feb413] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:01:56.458 WARN 2458 --- [ocp-async-4,4a92e85d0b603e71,03f3ec4305cd5f46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:56.458 INFO 2458 --- [ocp-async-1,1306d9a646ecf2ce,47cedd07c0a1952c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:56.707 INFO 2458 --- [prometheus-minute-schedule-1,a7cb2c038dd6de8f,f7011d8bea9658ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:56.715 INFO 2458 --- [prometheus-minute-schedule-1,a7cb2c038dd6de8f,f7011d8bea9658ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:56.875 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:56.876 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:01:57.530 INFO 2458 --- [ocp-schedules-13,a7c3c14de3d50358,29bfe94505a92579] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:01:57.539 INFO 2458 --- [ocp-schedules-13,a7c3c14de3d50358,29bfe94505a92579] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:01:57.585 INFO 2458 --- [ocp-schedules-13,a7c3c14de3d50358,29bfe94505a92579] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:01:58.458 INFO 2458 --- [ocp-async-3,4b5a791cb1bc2d20,48dbac52e536cf65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:01:58.458 WARN 2458 --- [ocp-async-9,c4d18515b755807d,5f125ea2b1b59a99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:01:58.504 INFO 2458 --- [ocp-async-2,6072313545787584,beba127efa2f76ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:01:59.068 INFO 2458 --- [ocp-schedules-16,a3e7a06bc39a8e73,7e42bec3a1723687] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:01:59.707 INFO 2458 --- [prometheus-minute-schedule-1,e633a11c65e70058,b7b63cd12fe68b8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534120, interval=60 2025-10-27 11:01:59.715 INFO 2458 --- [prometheus-minute-schedule-1,e633a11c65e70058,b7b63cd12fe68b8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:01:59.876 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:01:59.877 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:00.458 INFO 2458 --- [ocp-async-6,a74de225127f6d50,f888dbd4deea439a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:00.458 WARN 2458 --- [ocp-async-5,27d6ae92e1e06c54,19b4decc4a66c4f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:01.607 INFO 2458 --- [ocp-schedules-29,557c56515a5bc7b1,a6ec2f07be95b354] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:02:01.615 INFO 2458 --- [ocp-schedules-29,557c56515a5bc7b1,a6ec2f07be95b354] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:01.658 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:01.658 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:02:01.682 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:01.682 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:01.683 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:01.683 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:01.683 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:01.683 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:02:01.702 INFO 2458 --- [ocp-schedules-29,557c56515a5bc7b1,a6ec2f07be95b354] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:02:01.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:01.717520+08:00, lastProcessTime=2025-10-27T11:01:51.715929+08:00 2025-10-27 11:02:01.958 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:01.717520+08:00 2025-10-27 11:02:02.178 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b0974cfae5a29f01,fd3f46ce99be3843] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b0974cfae5a29f01, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:02:02.178 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b0974cfae5a29f01,fd3f46ce99be3843] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b0974cfae5a29f01, duration=0 ms] 2025-10-27 11:02:02.458 INFO 2458 --- [ocp-async-8,6f5c94879833bf71,4aed7fd02b22ab36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:02.458 WARN 2458 --- [ocp-async-1,f0e61797d6593fa5,aac6e02e1c936429] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:02.707 INFO 2458 --- [prometheus-minute-schedule-1,7fa2e5021a9b23a3,eb15b95e26d147bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:02.715 INFO 2458 --- [prometheus-minute-schedule-1,7fa2e5021a9b23a3,eb15b95e26d147bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:02.877 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:02.877 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:03.143 INFO 2458 --- [ocp-schedules-7,c42b5bbc61217d1e,64d11b1850e74dd3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:03.147 INFO 2458 --- [ocp-schedules-7,c42b5bbc61217d1e,64d11b1850e74dd3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:04.458 INFO 2458 --- [ocp-async-3,6000911b79e5c763,3781c11e07b8c8bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:04.458 WARN 2458 --- [ocp-async-2,5d9fdadab102a42d,5aaffa72f55b1ea7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:04.510 INFO 2458 --- [ocp-schedules-30,821b9197a71a5750,a96baa4d5259e6ed] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:04.552 WARN 2458 --- [scanner-112118117-1,821b9197a71a5750,ac58038deabb38a4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:04.700 INFO 2458 --- [alarm-detect-9,821b9197a71a5750,db514b651a1a25c9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534090 ~ 1761534114 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534105, value=69.531)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534105, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534105=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:02:05.583 INFO 2458 --- [ocp-schedules-8,358077353b6c7ad2,efed82cd4521279f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:05.707 INFO 2458 --- [prometheus-minute-schedule-1,dadaedae8573e60a,771a6453c4d47670] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:05.714 INFO 2458 --- [prometheus-minute-schedule-1,dadaedae8573e60a,771a6453c4d47670] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:05.877 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:05.878 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:02:06.052 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:02:06.098 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:02:06.127 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:02:06.127 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:02:06.127 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:02:06.127 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-27T10:59:30, target=2025-10-27T10:57:30, granule=120000ms, delay=120000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T10:57:30, target=2025-10-27T10:52:30, granule=600000ms, delay=300000ms) 2025-10-27 11:02:06.130 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=132ms, period=120000ms 2025-10-27 11:02:06.458 WARN 2458 --- [ocp-async-5,6d94fbc4be48cdbd,070455982d0f7787] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:06.458 INFO 2458 --- [ocp-async-7,ad8931bfd8bbee22,3ddfcc17af61c9fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:06.504 INFO 2458 --- [ocp-schedules-12,0c8d887f30610eb5,21491b16d3b38633] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:02:06.512 INFO 2458 --- [ocp-schedules-12,0c8d887f30610eb5,21491b16d3b38633] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:06.512 INFO 2458 --- [ocp-schedules-12,0c8d887f30610eb5,21491b16d3b38633] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:08.459 WARN 2458 --- [ocp-async-7,d4bff094adfac840,208919b548a31ea5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:08.459 INFO 2458 --- [ocp-async-1,fae14e8841db989a,38968dd0898d2771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:08.504 INFO 2458 --- [ocp-async-9,59bb5414efa139ff,0d6e39f1d0d5eafe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:08.707 INFO 2458 --- [prometheus-minute-schedule-1,69bc79de1f5dba61,93453cabe37aaf05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:08.717 INFO 2458 --- [prometheus-minute-schedule-1,69bc79de1f5dba61,93453cabe37aaf05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:08.878 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:08.879 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:10.009 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:02:10.011 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:02:10.047 INFO 2458 --- [Check obproxy1,ea2ebddd22150fa5,3e2678e984ea0461] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:02:10.047 INFO 2458 --- [Check obproxy1,ea2ebddd22150fa5,3e2678e984ea0461] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:02:10.048 INFO 2458 --- [Collect obproxy1,1f77c46f51062a6b,b215ad67918c58a8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:02:10.048 INFO 2458 --- [Collect obproxy1,1f77c46f51062a6b,b215ad67918c58a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:02:10.077 INFO 2458 --- [Check obproxy1,ea2ebddd22150fa5,3e2678e984ea0461] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:02:10.080 INFO 2458 --- [Collect obproxy1,1f77c46f51062a6b,b215ad67918c58a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:02:10.081 INFO 2458 --- [Check obproxy1,ea2ebddd22150fa5,3e2678e984ea0461] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:02:10.081 INFO 2458 --- [Check obproxy1,ea2ebddd22150fa5,3e2678e984ea0461] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:02:10.083 INFO 2458 --- [Collect obproxy1,1f77c46f51062a6b,b215ad67918c58a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:02:10.086 INFO 2458 --- [Collect obproxy1,1f77c46f51062a6b,b215ad67918c58a8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:02:10.179 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:02:10.459 INFO 2458 --- [ocp-async-9,e440007abdcbd2f9,985c63c58ecc07b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:10.459 WARN 2458 --- [ocp-async-2,8cebe5fab5e0e347,d9f77e2d89015483] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:11.707 INFO 2458 --- [prometheus-minute-schedule-1,dec1b3fd165217c4,e8d24d56c48f4fb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:11.716 INFO 2458 --- [prometheus-minute-schedule-1,dec1b3fd165217c4,e8d24d56c48f4fb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:11.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:11.717219+08:00, lastProcessTime=2025-10-27T11:02:01.717520+08:00 2025-10-27 11:02:11.879 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:11.880 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:11.931 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:11.717219+08:00 2025-10-27 11:02:12.003 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995} 2025-10-27 11:02:12.011 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClust 2025-10-27 11:02:12.011 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.013 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.013 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:02:12.013 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.017 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.017 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:02:12.017 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.020 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.029 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.053 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:02:12.053 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.163 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447732163000, endTimeUs=1761534011995000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:02:12.163Z, endTime=2025-10-27T03:00:11.995Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:02:12.459 INFO 2458 --- [ocp-async-9,5d09704a6828ffb9,cf242b44d21b6700] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:12.459 WARN 2458 --- [ocp-async-6,c9bd79ea87108b87,d6be25ab0909e2f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:12.507 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:02:12.507 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:02:12.537 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:02:12.538 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534011995000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:02:12.547 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:02:12.547 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:02:12.578 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:02:12.578 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534011995000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:02:12.583 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:02:12.583 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:02:12.163, end=2025-10-27T11:00:11.995}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:02:12.583 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.585 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.585 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:02:12.585 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.588 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:02:12.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.592 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:02:12.595 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:58:11.09, end=2025-10-27T11:00:11.995},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:02:12.595 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:02:12.595 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:02:12.601 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:02:13.065 INFO 2458 --- [ocp-schedules-4,b317eb8d7aa06eb5,bcb474d1cf645f5d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:02:13.065 INFO 2458 --- [ocp-schedules-4,b317eb8d7aa06eb5,bcb474d1cf645f5d] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:02:13.072 INFO 2458 --- [ocp-schedules-4,b317eb8d7aa06eb5,bcb474d1cf645f5d] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:02:13.072 INFO 2458 --- [ocp-schedules-4,b317eb8d7aa06eb5,bcb474d1cf645f5d] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:02:13.155 INFO 2458 --- [ocp-schedules-18,7ef50e342fd28a39,5f5a248edca53d1b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:13.158 INFO 2458 --- [ocp-schedules-18,7ef50e342fd28a39,5f5a248edca53d1b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:13.474 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534060, nodeCount=817, affectRows=817 2025-10-27 11:02:13.559 INFO 2458 --- [ocp-schedules-24,d52d4f4e8bc96971,4c42bd9d9df0927c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:02:13.559 INFO 2458 --- [ocp-schedules-24,d52d4f4e8bc96971,4c42bd9d9df0927c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:02:13.619 INFO 2458 --- [Collect all tenants major compactions1,d52d4f4e8bc96971,2c8e84fac114e42e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:02:13.620 INFO 2458 --- [Collect all tenants major compactions3,d52d4f4e8bc96971,ea982b2130855135] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:02:13.620 INFO 2458 --- [Collect all tenants major compactions2,d52d4f4e8bc96971,e6adabcb03378fa1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:02:13.620 INFO 2458 --- [Collect all tenants major compactions4,d52d4f4e8bc96971,9a9fe151290181ad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:02:13.623 INFO 2458 --- [Collect all tenants major compactions6,d52d4f4e8bc96971,c6c99720688156f5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:02:13.623 INFO 2458 --- [Collect all tenants major compactions5,d52d4f4e8bc96971,92214ce460894098] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:02:13.657 INFO 2458 --- [Collect all tenants major compactions6,d52d4f4e8bc96971,c6c99720688156f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.657 INFO 2458 --- [Collect all tenants major compactions5,d52d4f4e8bc96971,92214ce460894098] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.658 INFO 2458 --- [Collect all tenants major compactions6,d52d4f4e8bc96971,c6c99720688156f5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:02:13.658 INFO 2458 --- [Collect all tenants major compactions5,d52d4f4e8bc96971,92214ce460894098] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:02:13.662 INFO 2458 --- [Collect all tenants major compactions5,d52d4f4e8bc96971,92214ce460894098] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:02:13.663 INFO 2458 --- [Collect all tenants major compactions6,d52d4f4e8bc96971,c6c99720688156f5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:02:13.665 INFO 2458 --- [Collect all tenants major compactions2,d52d4f4e8bc96971,e6adabcb03378fa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.666 INFO 2458 --- [Collect all tenants major compactions2,d52d4f4e8bc96971,e6adabcb03378fa1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:02:13.668 INFO 2458 --- [Collect all tenants major compactions3,d52d4f4e8bc96971,ea982b2130855135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.669 INFO 2458 --- [Collect all tenants major compactions3,d52d4f4e8bc96971,ea982b2130855135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:02:13.670 INFO 2458 --- [Collect all tenants major compactions1,d52d4f4e8bc96971,2c8e84fac114e42e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.671 INFO 2458 --- [Collect all tenants major compactions1,d52d4f4e8bc96971,2c8e84fac114e42e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:02:13.672 INFO 2458 --- [Collect all tenants major compactions2,d52d4f4e8bc96971,e6adabcb03378fa1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:02:13.673 INFO 2458 --- [Collect all tenants major compactions3,d52d4f4e8bc96971,ea982b2130855135] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:02:13.675 INFO 2458 --- [Collect all tenants major compactions1,d52d4f4e8bc96971,2c8e84fac114e42e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:02:13.679 INFO 2458 --- [Collect all tenants major compactions4,d52d4f4e8bc96971,9a9fe151290181ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:13.680 INFO 2458 --- [Collect all tenants major compactions4,d52d4f4e8bc96971,9a9fe151290181ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:02:13.683 INFO 2458 --- [Collect all tenants major compactions4,d52d4f4e8bc96971,9a9fe151290181ad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:02:13.684 INFO 2458 --- [ocp-schedules-24,d52d4f4e8bc96971,4c42bd9d9df0927c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:02:13.962 INFO 2458 --- [ocp-schedules-25,b22e34767b3693e0,9ad1ffd7c1c4b7b4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:02:13.962+08:00, clusters=[1, 2] 2025-10-27 11:02:13.965 INFO 2458 --- [ocp-schedules-25,b22e34767b3693e0,9ad1ffd7c1c4b7b4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:02:13.967 INFO 2458 --- [ocp-schedules-25,b22e34767b3693e0,9ad1ffd7c1c4b7b4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:02:14.459 INFO 2458 --- [ocp-async-9,8f5844a5227dd044,7fcb0f0c5c13463a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:14.459 WARN 2458 --- [ocp-async-8,c613c12d2ffb0b59,00e471b4b6070976] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:14.707 INFO 2458 --- [ocp-schedules-26,1e927c22b92f370b,40f4671b6e922cad] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:14.707 INFO 2458 --- [prometheus-minute-schedule-1,028d7e9a61b8dcdb,33054c02b181ca7c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:14.722 INFO 2458 --- [prometheus-minute-schedule-1,028d7e9a61b8dcdb,33054c02b181ca7c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:14.790 WARN 2458 --- [scanner-112119951-1,1e927c22b92f370b,848ad30682995529] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:14.840 INFO 2458 --- [alarm-detect-6,1e927c22b92f370b,6980e0159097bab8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534105 ~ 1761534124 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534120, value=96.135)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534120, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534120=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:02:14.875 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587936) Duplicate entry '835370603008958464-1761534000' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587936) Duplicate entry '835370603008958464-1761534000' for key 'PRIMARY' 2025-10-27 11:02:14.880 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:14.881 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:15.591 INFO 2458 --- [ocp-schedules-24,27debc5e55c3fba9,aaf5fbb0a58d0fbd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:15.734 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534060, nodeCount=1000, affectRows=1000 2025-10-27 11:02:15.738 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534060, nodeCount=994, validCount=992 2025-10-27 11:02:16.459 WARN 2458 --- [ocp-async-9,cc2348b628fadfc6,ca9b5dbf54df71ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:16.459 INFO 2458 --- [ocp-async-5,0bbff7c4f574d101,78429a9493ab787d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:17.707 INFO 2458 --- [prometheus-minute-schedule-1,0b4585386f03ef8a,29462867bb3c2754] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:17.718 INFO 2458 --- [prometheus-minute-schedule-1,0b4585386f03ef8a,29462867bb3c2754] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:17.779 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534060, nodeCount=1000, affectRows=1000 2025-10-27 11:02:17.782 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534060, nodeCount=999, validCount=995 2025-10-27 11:02:17.881 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:17.882 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:17.909 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534060, nodeCount=74, affectRows=74 2025-10-27 11:02:17.910 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534060, nodeCount=73, validCount=73 2025-10-27 11:02:18.459 INFO 2458 --- [ocp-async-6,499f14adf63be013,490caaedb457c218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:18.459 WARN 2458 --- [ocp-async-3,e9b6249c210de7b8,e68a9c1d9b9bebf0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:18.505 INFO 2458 --- [ocp-async-8,9f31d0ceb776240b,1d95eeb44e61603c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:18.650 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534060, nodeCount=1000, affectRows=1000 2025-10-27 11:02:18.655 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534060, nodeCount=1000, validCount=996 2025-10-27 11:02:18.679 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534060, nodeCount=1000, affectRows=1000 2025-10-27 11:02:18.684 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534060, nodeCount=996, validCount=992 2025-10-27 11:02:18.759 INFO 2458 --- [ocp-schedules-18,e37e23472e118926,8eefcd5c49fc2368] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:18.759 INFO 2458 --- [ocp-schedules-18,e37e23472e118926,8eefcd5c49fc2368] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:02:18.855 INFO 2458 --- [ocp-schedules-21,d86b5aafde41deff,6afa1870a81fe1d2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:02:20.459 INFO 2458 --- [ocp-async-8,9edd7ddd28d9cc7a,6f4bfb3c50d47367] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:20.460 WARN 2458 --- [ocp-async-1,0f0d9b9b74d97c24,10bbf140953b4031] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:20.707 INFO 2458 --- [prometheus-minute-schedule-1,ba1813bdc5dd14ff,0d8a11d022f913d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:20.718 INFO 2458 --- [prometheus-minute-schedule-1,ba1813bdc5dd14ff,0d8a11d022f913d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:20.882 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:20.882 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:21.317 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:21.317 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2025-10-27 11:02:21.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2025-10-27 11:02:21.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2025-10-27 11:02:21.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2025-10-27 11:02:21.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2025-10-27 11:02:21.336 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : binlogCluster BinlogCluster(id=4, name=binlogtest2, version=4.0.1-100000162024122511, status=RUNNING, metadbInfo=BinlogClusterMetadbInfo(tenantId=2, username=root, database=binlog_database, tenantName=ocp_meta, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187, clusterType=PRIMARY, obVersion=4.3.5.4, address=10.40.65.146, port=2881, fullUserName=root@ocp_meta), resourceStats=null, servers=[BinlogServer(id=4, ip=10.40.65.151, port=2983, binlogClusterId=4, binlogClusterName=binlogtest2, hostId=4, hostName=tjtx40-65-151.zhuanos.com, hostKind=DEDICATED_PHYSICAL_MACHINE, hostTypeName=test, idcName=default_idc, regionName=deault_region, status=RUNNING, resourceStats=null)], instances=null, tags=null, createTime=2025-10-24T17:01:22+08:00, updateTime=2025-10-24T17:04:41+08:00, installPath=/home/admin/binlogservice, binlogFilePath=/home/admin/binlogservice/run, address=10.40.65.151, port=2983) 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.400 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.400 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:02:21.416 INFO 2458 --- [ocp-schedules-17,ff3a569ff9e04f7a,3265a11d80a26d5f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:21.420 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:02:21.421 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:02:21.421 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:02:21.421 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:02:21.421 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:02:21.422 INFO 2458 --- [ocp-schedules-15,dd7480cc9d67cd8a,a26c00e437522e49] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:21.428 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:02:21.429 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:02:21.429 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:02:21.429 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:02:21.430 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:02:21.440 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:21.442 INFO 2458 --- [ocp-schedules-9,583520a949a72ad5,aef81b53add513cf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:02:21.442 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2025-10-27 11:02:21.442 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:21.442 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:21.442 INFO 2458 --- [ocp-schedules-12,53a283da74eff8da,41936435352d2f63] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:21.443 INFO 2458 --- [host-check-clockdiff1,53a283da74eff8da,cc7005ca284d19f5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:02:21.443 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:21.444 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:21.444 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:21.445 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:21.446 INFO 2458 --- [host-check-clockdiff2,53a283da74eff8da,1242d3f72f95be10] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:02:21.447 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:21.447 INFO 2458 --- [host-check-clockdiff3,53a283da74eff8da,27fe2933e22d341a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:02:21.447 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:21.447 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:21.448 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:21.448 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:21.449 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:21.449 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:21.450 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:21.450 INFO 2458 --- [host-check-clockdiff4,53a283da74eff8da,7a75c4efe3a1cdec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:02:21.450 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:21.451 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:21.452 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:21.452 INFO 2458 --- [host-check-clockdiff5,53a283da74eff8da,b278578b9fe4bd08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:02:21.453 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:21.455 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:21.478 INFO 2458 --- [host-check-clockdiff3,53a283da74eff8da,27fe2933e22d341a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534141 0 0 , error= 2025-10-27 11:02:21.483 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:21.497 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:21.497 INFO 2458 --- [host-check-clockdiff5,53a283da74eff8da,b278578b9fe4bd08] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534141 0 0 , error= 2025-10-27 11:02:21.498 INFO 2458 --- [host-check-clockdiff4,53a283da74eff8da,7a75c4efe3a1cdec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534141 0 0 , error= 2025-10-27 11:02:21.500 INFO 2458 --- [host-check-clockdiff1,53a283da74eff8da,cc7005ca284d19f5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534141 0 0 , error= 2025-10-27 11:02:21.504 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:02:21.504 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:02:21.504 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:02:21.511 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2025-10-27 11:02:21.517 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:02:21.517 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:02:21.517 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:02:21.519 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:02:21.519 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:02:21.519 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:02:21.522 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2025-10-27 11:02:21.524 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2025-10-27 11:02:21.531 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2025-10-27 11:02:21.546 INFO 2458 --- [ocp-schedules-27,e5a49aff072f150d,e6c70ce2979f2be0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:02:21.547 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:02:21.549 INFO 2458 --- [ocp-schedules-22,d3e62d3b4814706c,6d3e47fc718919e3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:02:21.551 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2 to set sql collect property inactive 2025-10-27 11:02:21.551 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:02:21.551 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:02:21.551 INFO 2458 --- [host-check-agent-status5,ff3a569ff9e04f7a,e45903fc48d644e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:02:21.552 INFO 2458 --- [ocp-schedules-27,e5a49aff072f150d,e6c70ce2979f2be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:21.552 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:02:21.553 INFO 2458 --- [ocp-schedules-22,d3e62d3b4814706c,6d3e47fc718919e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:21.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:21.553 INFO 2458 --- [ocp-schedules-22,d3e62d3b4814706c,6d3e47fc718919e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:02:21.553 INFO 2458 --- [ocp-schedules-22,485af26b4fe1e76c,72d2c540d3f0e0ec] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:02:21.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2025-10-27 11:02:21.554 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:02:21.556 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:21.557 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:02:21.557 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:02:21.557 INFO 2458 --- [host-check-agent-status2,ff3a569ff9e04f7a,99cc3cd032cd3873] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:02:21.557 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:02:21.560 INFO 2458 --- [ocp-schedules-22,485af26b4fe1e76c,72d2c540d3f0e0ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:21.560 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:21.561 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:21.562 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:02:21.562 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:02:21.562 INFO 2458 --- [host-check-agent-status3,ff3a569ff9e04f7a,9eb583125d65d4e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:02:21.564 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:02:21.565 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:02:21.564462396+08:00 2025-10-27 11:02:21.566 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 3 to set sql collect property inactive 2025-10-27 11:02:21.566 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:02:21.566 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:02:21.566 INFO 2458 --- [host-check-agent-status4,ff3a569ff9e04f7a,97269303760e9ab8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:02:21.566 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:02:21.565+08:00 2025-10-27 11:02:21.566 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:21.567 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:02:21.567 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:02:21.568 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:02:21.567741708+08:00 2025-10-27 11:02:21.568 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:02:21.567670091+08:00 2025-10-27 11:02:21.569 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:02:21.568+08:00 2025-10-27 11:02:21.569 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:02:21.568+08:00 2025-10-27 11:02:21.569 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:21.569 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:02:21.570 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:21.573 INFO 2458 --- [ocp-schedules-2,bf053144cc6f966e,7d2f4bc557833593] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:02:21.574 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:02:21.574 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:02:21.574 INFO 2458 --- [host-check-agent-status1,ff3a569ff9e04f7a,6d099dc6b37d46ac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:02:21.576 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.binlog.cluster.id=4), (monagent.binlog.cluster.name=binlogtest2), (monagent.pipeline.logproxy.status=active), (binlog.multi.instance.config={"instanceConfigs":[],"instanceType":"binlog"}), (binlog.install.path=/home/admin/binlogservice), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_hardware_error","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["Hardware Error"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (mo 2025-10-27 11:02:21.576 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:21.583 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:02:21.565+08:00 and status AVAILABLE 2025-10-27 11:02:21.586 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:02:21.568+08:00 and status AVAILABLE 2025-10-27 11:02:21.589 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:02:21.568+08:00 and status AVAILABLE 2025-10-27 11:02:21.589 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:02:21.590 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:02:21.591 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:02:21.591054549+08:00 2025-10-27 11:02:21.591 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:02:21.591+08:00 2025-10-27 11:02:21.593 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:02:21.596 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:02:21.598 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:02:21.599 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:02:21.598717248+08:00 2025-10-27 11:02:21.600 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:02:21.591+08:00 and status AVAILABLE 2025-10-27 11:02:21.600 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:02:21.599+08:00 2025-10-27 11:02:21.604 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.604 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:02:21.605 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:21.606 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:02:21.610 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2025-10-27 11:02:21.611 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:02:21.599+08:00 and status AVAILABLE 2025-10-27 11:02:21.616 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:02:21.627 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:21.631 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:21.632 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:21.635 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:21.636 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:21.637 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:21.639 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:21.644 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:21.654 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510188), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.66.107), (ob.run.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.66.107:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase2), (ob.logtailer.enabled=true), (ob.l 2025-10-27 11:02:21.656 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:21.656 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.657 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:21.660 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:21.663 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:02:21.664 INFO 2458 --- [ocp-schedules-27,e5a49aff072f150d,e6c70ce2979f2be0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:02:21.670 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.670 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:21.670 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.146), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.146:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:02:21.682 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.67.22), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.67.22:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtailer. 2025-10-27 11:02:21.683 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.168), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.168:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:02:21.684 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2025-10-27 11:02:21.704 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2025-10-27 11:02:21.713 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2025-10-27 11:02:21.715 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.715 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:21.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:21.718529+08:00, lastProcessTime=2025-10-27T11:02:11.717219+08:00 2025-10-27 11:02:21.720 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2025-10-27 11:02:21.723 INFO 2458 --- [ocp-schedules-11,37e334f9a242ff7d,91b4b7b2f16385b3] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-10-27 11:02:21.733 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:21.749 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:21.749 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:21.758 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:21.759 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:02:21.764 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:02:21.768 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:02:21.780 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:02:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:02:21.792 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:02:21.797 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:21.801 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:02:21.802 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:02:21.803 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:21.804 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.804 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:02:21.810 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:02:21.827 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:02:21.850 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:21.855 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:21.858 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:02:21.870 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:21.873 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:02:21.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:21.880 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:21.882 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.885 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:02:21.886 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:21.887 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:02:21.893 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:02:21.896 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:02:21.906 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:02:21.912 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:21.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:21.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:21.926 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:02:21.929 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:21.930 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:02:21.930 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.931 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:21.932 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:02:21.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:21.942 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:02:21.947 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:21.974 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:02:21.975 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:21.992 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:02:21.997 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:21.998 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:21.999 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:21.999 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:02:22.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:22.006 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:22.007 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:22.012 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:02:22.013 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:02:22.021 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:02:22.022 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:02:22.024 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:22.024 INFO 2458 --- [host-check-host-status5,dd7480cc9d67cd8a,ebbd08689ee7700a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:02:22.031 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:22.031 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:02:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:22.056 INFO 2458 --- [host-check-host-status4,dd7480cc9d67cd8a,76bf77ae7ff4b893] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:02:22.061 INFO 2458 --- [host-check-host-status2,dd7480cc9d67cd8a,ea82760601f6e58c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:02:22.061 INFO 2458 --- [host-check-host-status3,dd7480cc9d67cd8a,9f512cbdc4fa8653] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:02:22.067 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:22.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.093 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:22.094 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:02:22.097 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:02:22.124 INFO 2458 --- [host-check-host-status1,dd7480cc9d67cd8a,6ad6d1bddacafded] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:02:22.137 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:02:22.137 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.137 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.137 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:22.138 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:02:22.139 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.140 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:02:22.158 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:02:22.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:22.183 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:22.183 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.184 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:02:22.209 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:02:22.209 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:02:22.215 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:22.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:22.253 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.254 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:22.258 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:02:22.267 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:02:22.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:22.280 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:22.334 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:02:22.406 INFO 2458 --- [host-check-clockdiff2,53a283da74eff8da,1242d3f72f95be10] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534142 -1 -1 , error= 2025-10-27 11:02:22.408 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:02:22.408 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.409 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:02:22.411 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.411 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:02:22.443 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:02:22.460 INFO 2458 --- [ocp-async-9,d982b4b5be760b8c,e2333a08d44a385a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:22.460 WARN 2458 --- [ocp-async-10,76ef214954c4b33e,b7825b4b450c0528] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:22.460 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:21.718529+08:00 2025-10-27 11:02:22.484 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:22.484 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:22.485 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:02:22.524 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:02:22.524 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:02:22.524 INFO 2458 --- [ocp-schedules-22,485af26b4fe1e76c,72d2c540d3f0e0ec] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:02:23.176 INFO 2458 --- [ocp-schedules-11,b4a364baf1dd1b91,b6751215eb924cc1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:23.179 INFO 2458 --- [ocp-schedules-11,b4a364baf1dd1b91,b6751215eb924cc1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:23.706 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:02:23.706 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:02:23.707 INFO 2458 --- [prometheus-minute-schedule-1,647f7a87167341a5,37f212693bc3013f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:23.710 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:02:23.710 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:02:23.710 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:02:23.710 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:02:23.714 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:02:23.714 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:02:23.714 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:02:23.714 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:02:23.719 INFO 2458 --- [prometheus-minute-schedule-1,647f7a87167341a5,37f212693bc3013f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:23.739 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:02:23.744 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:02:23.744 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:02:23.744 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:02:23.744 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:02:23.744 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:02:23.750 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:02:23.882 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:23.883 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:24.106 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:02:24.460 INFO 2458 --- [ocp-async-3,bf590b3029735715,c756e0ddb75e1159] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:24.460 WARN 2458 --- [ocp-async-5,5134e8e3df4b642b,31ca9186e6e281e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:24.492 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.777955093S 2025-10-27 11:02:24.492 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:02:24.560 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:02:24.560 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:02:24.594 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:02:24.641 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:02:24.651 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:02:24.770 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] o.s.boot.SpringApplication : Started application in 1.059 seconds (JVM running for 256613.197) 2025-10-27 11:02:24.770 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.059725985S 2025-10-27 11:02:24.791 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [4012]; (conn=134587936) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again.; nested exception is java.sql.BatchUpdateException: (conn=134587936) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1576) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:691) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1034) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1088) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=134587936) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1048) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=134587936) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2119) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-10-27 11:02:24.845 INFO 2458 --- [ocp-schedules-8,f5b22f98f51958bc,ef62f9db54d01262] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:25.154 WARN 2458 --- [scanner-112120116-1,f5b22f98f51958bc,a2f6ae7f8f21ba23] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:25.168 INFO 2458 --- [ocp-schedules-14,14dc1414a4d74eef,0c5008bfac4cd98e] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:02:25.340 INFO 2458 --- [alarm-detect-10,f5b22f98f51958bc,b9197976003b1cfe] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534120 ~ 1761534135 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534135, value=91.659)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534135, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534135=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:02:25.600 INFO 2458 --- [ocp-schedules-1,5c16685b96107099,1e9ae5c6eeaf6d2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:26.460 INFO 2458 --- [ocp-async-3,00d629ef1bd17000,e8a46066d1c9ec24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:26.460 WARN 2458 --- [ocp-async-6,c8faae3f6937de08,2ff4d8d3ced8b3e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:26.515 INFO 2458 --- [ocp-schedules-26,9e611a7845dc8c53,a2ea0fabbc0e43aa] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:02:26.523 INFO 2458 --- [ocp-schedules-26,9e611a7845dc8c53,a2ea0fabbc0e43aa] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:26.523 INFO 2458 --- [ocp-schedules-26,9e611a7845dc8c53,a2ea0fabbc0e43aa] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:26.707 INFO 2458 --- [prometheus-minute-schedule-1,cde8ceaa0e557a03,abecec315838a2a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:26.713 INFO 2458 --- [prometheus-minute-schedule-1,cde8ceaa0e557a03,abecec315838a2a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:26.883 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:26.884 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:28.460 WARN 2458 --- [ocp-async-4,5cea8902c1defad9,eaf96d5b7507dde8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:28.460 INFO 2458 --- [ocp-async-3,c127147f2a2b6efd,e95109cb9d86ea16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:28.505 INFO 2458 --- [ocp-async-10,698f4d43d7db158a,ddb9d0aba4506de5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:28.777 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:02:28.777 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:02:28.782 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:02:28.782 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:02:28.786 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:02:28.786 INFO 2458 --- [ocp-schedules-22,31c14da819e49878,aeeadacb2f21a276] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:02:29.276 INFO 2458 --- [ocp-schedules-18,1492c6b3553e7a4d,42f4e24cfa0e1b24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:02:29.281 INFO 2458 --- [ocp-schedules-18,1492c6b3553e7a4d,42f4e24cfa0e1b24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:02:29.281 INFO 2458 --- [ocp-schedules-18,1492c6b3553e7a4d,42f4e24cfa0e1b24] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:02:29.707 INFO 2458 --- [prometheus-minute-schedule-1,a7d65502e0be8d25,2da9d8c4720b2d79] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:29.716 INFO 2458 --- [prometheus-minute-schedule-1,a7d65502e0be8d25,2da9d8c4720b2d79] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:29.884 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:29.884 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:30.460 WARN 2458 --- [ocp-async-5,e6d5b22693203e4b,45d792a27b908459] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:30.460 INFO 2458 --- [ocp-async-9,3d2e86a8ff95e4b9,d0748c367b6c3e92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:31.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:31.716823+08:00, lastProcessTime=2025-10-27T11:02:21.718529+08:00 2025-10-27 11:02:31.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:31.716823+08:00 2025-10-27 11:02:32.460 INFO 2458 --- [ocp-async-9,55c9d5c7a8b62bae,f12f44666678ffd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:32.460 WARN 2458 --- [ocp-async-1,515ad5bae71dc6e1,708ae71a82784a3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:32.707 INFO 2458 --- [prometheus-minute-schedule-1,cea68411a5b4e990,03bcb8c74240f8b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:32.716 INFO 2458 --- [prometheus-minute-schedule-1,cea68411a5b4e990,03bcb8c74240f8b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:32.884 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:32.886 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:33.189 INFO 2458 --- [ocp-schedules-1,9530fe8298ed02ef,6d626953e5cad409] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:33.193 INFO 2458 --- [ocp-schedules-1,9530fe8298ed02ef,6d626953e5cad409] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:33.690 INFO 2458 --- [ocp-schedules-25,9029a2b24c56f084,4c7efb36480b455a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:02:33.690 INFO 2458 --- [ocp-schedules-25,9029a2b24c56f084,4c7efb36480b455a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:02:33.936 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:02:33.936 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:02:33.989 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=root 2025-10-27 11:02:33.989 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:33.989 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:34.037 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:34.037 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:34.037 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:34.038 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:02:34.050 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:34.050 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:34.051 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:34.051 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:02:34.083 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:02:34.106 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:34.107 INFO 2458 --- [ocp-schedules-13,9532d5d6180ace68,0eec196e14a0ff01] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:02:34.460 INFO 2458 --- [ocp-async-9,f6b874e31ccc56bd,3b742902a6f2fa0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:34.460 WARN 2458 --- [ocp-async-8,061d2bcdc1952bc8,aaf38c3b75bd5a8e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:35.350 INFO 2458 --- [ocp-schedules-25,f2a7e3d030405b43,8e7680d37f2127a4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:35.393 WARN 2458 --- [scanner-112120336-1,f2a7e3d030405b43,a9ac561866f55d0a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:35.608 INFO 2458 --- [ocp-schedules-7,33f430fa6729a507,4e8c94bfe0a3704e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:35.707 INFO 2458 --- [prometheus-minute-schedule-1,bfadabf573084569,f4e0b5f98dc2cb69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:35.714 INFO 2458 --- [prometheus-minute-schedule-1,bfadabf573084569,f4e0b5f98dc2cb69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:35.886 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:35.887 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:36.436 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:36.436 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:02:36.436 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:02:36.436 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:02:36.436 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:02:36.437 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:02:36.450 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:36.450 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:36.451 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:36.452 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:36.452 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:36.455 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:36.455 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:36.455 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:36.456 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:36.456 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:36.461 INFO 2458 --- [ocp-async-9,2a699b4550d646f3,b6e751ba01a1e9f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:36.461 WARN 2458 --- [ocp-async-7,08086c2eedf45da0,0386c3fbe21f23f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:36.519 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:02:36.519 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:02:36.519 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:02:36.520 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:02:36.531 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:02:36.533 INFO 2458 --- [ocp-schedules-11,9236a69d59970d7e,61a51c14d9acbbd3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:02:37.708 INFO 2458 --- [prometheus-cache-monitor1,501a90a31e43e1bc,fe78148c3df78a34] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20295,"minuteWriteQueue":0,"minuteReadCacheSize":7851,"secondWriteCacheSize":5074,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":15714} 2025-10-27 11:02:37.708 INFO 2458 --- [prometheus-cache-monitor1,501a90a31e43e1bc,fe78148c3df78a34] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:02:38.461 WARN 2458 --- [ocp-async-4,fe0326237e0d36f2,cb2fcab6e290c40f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:38.461 INFO 2458 --- [ocp-async-9,b662cb4a391b2138,b21c4fc200ab154f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:38.505 INFO 2458 --- [ocp-async-8,32141b866ac74823,f421a097eeb80c2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:38.550 INFO 2458 --- [ocp-schedules-3,a1c306ad15a0103d,fb3aae7ec4655caf] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:02:38.556 INFO 2458 --- [ocp-schedules-3,a1c306ad15a0103d,fb3aae7ec4655caf] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:38.556 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:02:38.557 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:02:38.557 INFO 2458 --- [ocp-schedules-3,a1c306ad15a0103d,fb3aae7ec4655caf] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:02:38.707 INFO 2458 --- [prometheus-minute-schedule-1,358b6a6f906e23db,41816d3ab72274ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:38.721 INFO 2458 --- [prometheus-minute-schedule-1,358b6a6f906e23db,41816d3ab72274ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:38.887 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:38.888 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:40.013 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:02:40.013 INFO 2458 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:02:40.051 INFO 2458 --- [Collect obproxy1,cec29db83f757b27,e373b58c80e08e83] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:02:40.051 INFO 2458 --- [Check obproxy1,ebe474b0b66eb705,6985d3f1f73dac67] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:02:40.055 INFO 2458 --- [Collect obproxy1,cec29db83f757b27,e373b58c80e08e83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:02:40.057 INFO 2458 --- [Check obproxy1,ebe474b0b66eb705,6985d3f1f73dac67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:02:40.092 INFO 2458 --- [Check obproxy1,ebe474b0b66eb705,6985d3f1f73dac67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:02:40.092 INFO 2458 --- [Collect obproxy1,cec29db83f757b27,e373b58c80e08e83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:02:40.095 INFO 2458 --- [Check obproxy1,ebe474b0b66eb705,6985d3f1f73dac67] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:02:40.095 INFO 2458 --- [Check obproxy1,ebe474b0b66eb705,6985d3f1f73dac67] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:02:40.096 INFO 2458 --- [Collect obproxy1,cec29db83f757b27,e373b58c80e08e83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:02:40.099 INFO 2458 --- [Collect obproxy1,cec29db83f757b27,e373b58c80e08e83] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:02:40.122 INFO 2458 --- [ocp-schedules-19,095db3c671e4bfff,3c0c517cf68e4006] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:02:40.123 INFO 2458 --- [ocp-schedules-19,095db3c671e4bfff,3c0c517cf68e4006] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:02:40.461 INFO 2458 --- [ocp-async-2,4fac45b72276f07c,349406c95e60aa3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:40.461 WARN 2458 --- [ocp-async-3,7490aa786f1e4eef,716c99aa3306387a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:41.707 INFO 2458 --- [prometheus-minute-schedule-1,d589b3fd4ac03b3f,fca7988f01ff0f21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:41.715 INFO 2458 --- [prometheus-minute-schedule-1,d589b3fd4ac03b3f,fca7988f01ff0f21] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:41.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:41.718430+08:00, lastProcessTime=2025-10-27T11:02:31.716823+08:00 2025-10-27 11:02:41.888 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:41.889 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:41.938 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:41.718430+08:00 2025-10-27 11:02:42.461 INFO 2458 --- [ocp-async-2,662aa532276f2042,db799efd34a5eb69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:42.461 WARN 2458 --- [ocp-async-10,ebf62c44675d7967,6788fb87e31a4e2d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:43.203 INFO 2458 --- [ocp-schedules-15,8c52dd978bc52bf9,2364ff392f7ea7f2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:43.217 INFO 2458 --- [ocp-schedules-17,6cc82974e9a51278,274bf1bce8ea6205] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:02:43.217 INFO 2458 --- [ocp-schedules-17,6cc82974e9a51278,274bf1bce8ea6205] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:02:43.224 INFO 2458 --- [ocp-schedules-15,8c52dd978bc52bf9,2364ff392f7ea7f2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:43.225 INFO 2458 --- [ocp-schedules-17,6cc82974e9a51278,274bf1bce8ea6205] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:02:43.225 INFO 2458 --- [ocp-schedules-17,6cc82974e9a51278,274bf1bce8ea6205] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:02:43.579 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,48916e81a11f3980,cb3bf8d3f912b936] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=48916e81a11f3980, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:02:43.579 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,48916e81a11f3980,cb3bf8d3f912b936] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=48916e81a11f3980, duration=1 ms] 2025-10-27 11:02:43.931 INFO 2458 --- [ocp-schedules-23,b3f466642a400419,d44d37226f52204e] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:02:44.461 INFO 2458 --- [ocp-async-1,4b2dbb7136259733,ab1c7d2e9dd507c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:44.461 WARN 2458 --- [ocp-async-6,339109bb2310e1bd,b59964ab67958b78] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:44.707 INFO 2458 --- [prometheus-minute-schedule-1,df08a7028b428afe,11073aab2a427132] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:44.715 INFO 2458 --- [prometheus-minute-schedule-1,df08a7028b428afe,11073aab2a427132] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:44.889 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:44.890 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:45.102 INFO 2458 --- [prometheus-second-schedule-1,e2c89e56006b2382,70dcf600871ec93c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:02:45.492 INFO 2458 --- [ocp-schedules-25,4d1e85789149b986,1b2466da8fc5e0da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:45.543 WARN 2458 --- [scanner-112120538-1,4d1e85789149b986,cffdf02f9d199cb2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:45.584 INFO 2458 --- [alarm-detect-15,4d1e85789149b986,5ae94b703b3a92b6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534095 ~ 1761534155 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534120, value=202454.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202454.56, values={1761534120=202454.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:02:45.585 INFO 2458 --- [alarm-detect-15,4d1e85789149b986,5ae94b703b3a92b6] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:02:45.615 INFO 2458 --- [ocp-schedules-3,bcb5eb8ef6eee19a,b69d7cc926dc9583] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:45.615 INFO 2458 --- [alarm-detect-2,4d1e85789149b986,497b2716ce60dcb1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534095 ~ 1761534155 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534120, value=92.925)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.925, values={1761534120=92.925}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:02:45.616 INFO 2458 --- [alarm-detect-2,4d1e85789149b986,497b2716ce60dcb1] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:02:45.668 INFO 2458 --- [alarm-detect-5,4d1e85789149b986,6f039226fb9a5677] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534095 ~ 1761534155 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534120, value=202455.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202455.912, values={1761534120=202455.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:02:45.669 INFO 2458 --- [alarm-detect-5,4d1e85789149b986,6f039226fb9a5677] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:02:45.676 INFO 2458 --- [alarm-detect-2,4d1e85789149b986,297f28a1f2591c60] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534135 ~ 1761534155 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534150, value=61.826)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534150, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534150=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:02:46.037 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:02:46.064 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:46.069 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:46.156 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:46.160 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:46.219 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:46.223 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:46.283 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:46.286 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:46.352 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:46.355 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:46.410 INFO 2458 --- [ocp-schedules-29,78cb353902b89991,c513b0b1c213b390] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:02:46.461 INFO 2458 --- [ocp-async-3,36e9a82af077801f,a725b9ba94d0541e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:46.461 WARN 2458 --- [ocp-async-8,acd4919222f32daa,43151f6ee24ee670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:46.526 INFO 2458 --- [ocp-schedules-8,c3118b83d267ca77,d192f47633952ddd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:02:46.534 INFO 2458 --- [ocp-schedules-8,c3118b83d267ca77,d192f47633952ddd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:46.534 INFO 2458 --- [ocp-schedules-8,c3118b83d267ca77,d192f47633952ddd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:02:47.310 INFO 2458 --- [ocp-schedules-1,9469243382ce36b5,6b799408429a165b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:02:47.310+08:00, clusters=[1, 2] 2025-10-27 11:02:47.341 INFO 2458 --- [ocp-schedules-1,9469243382ce36b5,6b799408429a165b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:02:47.344 INFO 2458 --- [ocp-schedules-1,9469243382ce36b5,6b799408429a165b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:02:47.707 INFO 2458 --- [prometheus-minute-schedule-1,541cb9f867df3a1e,daea70d90175b410] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:47.715 INFO 2458 --- [prometheus-minute-schedule-1,541cb9f867df3a1e,daea70d90175b410] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:47.890 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:47.891 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:48.461 INFO 2458 --- [ocp-async-10,5e65e2767a2da10f,16176e3c130b0461] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:48.461 WARN 2458 --- [ocp-async-5,924e4a4dda30a1fa,84bebcde72d552bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:48.505 INFO 2458 --- [ocp-async-6,ce9e03c4b5187f4b,45fc14135bf4bbc8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:50.462 INFO 2458 --- [ocp-async-2,a776c5e14c6ede65,7693ae4b33bb790a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:50.462 WARN 2458 --- [ocp-async-1,077d0bce30c6cb99,b3f4203008808fcb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:50.707 INFO 2458 --- [prometheus-minute-schedule-1,04f5716aa0ca89d3,635d896c33a9113f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:50.716 INFO 2458 --- [prometheus-minute-schedule-1,04f5716aa0ca89d3,635d896c33a9113f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:50.891 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:50.891 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:51.412 INFO 2458 --- [ocp-schedules-20,748f69abcdd9ad27,ee6b5dc8a626fea6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:51.412 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:02:51.413 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:02:51.413 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:02:51.413 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:02:51.413 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:02:51.416 INFO 2458 --- [ocp-schedules-6,d7ea64c0537586c3,70b6400e4e6cf607] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:51.416 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:02:51.416 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:02:51.417 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:02:51.417 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:02:51.417 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:02:51.427 INFO 2458 --- [ocp-schedules-26,f9bcdaaa9aab3da3,2df7028ae7db2788] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:02:51.428 INFO 2458 --- [host-check-clockdiff1,f9bcdaaa9aab3da3,d84fc156b44c3719] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:02:51.428 INFO 2458 --- [host-check-clockdiff3,f9bcdaaa9aab3da3,2725042bbe13bc67] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:02:51.428 INFO 2458 --- [host-check-clockdiff5,f9bcdaaa9aab3da3,66e9a0fcc0868186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:02:51.428 INFO 2458 --- [host-check-clockdiff4,f9bcdaaa9aab3da3,8c91a3fc2db95bf6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:02:51.428 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:51.429 INFO 2458 --- [host-check-clockdiff2,f9bcdaaa9aab3da3,c337c079f2537364] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:02:51.429 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:51.429 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:51.429 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:51.430 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:51.434 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:51.434 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:51.441 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:51.441 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:51.442 INFO 2458 --- [ocp-schedules-25,8e9145def93affcc,c5d3a924b1805177] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:02:51.442 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:51.442 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:51.442 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:51.443 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:51.443 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:51.443 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:51.449 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:51.449 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:51.449 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:51.450 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:51.452 INFO 2458 --- [host-check-clockdiff2,f9bcdaaa9aab3da3,c337c079f2537364] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534171 -1 -1 , error= 2025-10-27 11:02:51.454 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:51.454 INFO 2458 --- [host-check-clockdiff3,f9bcdaaa9aab3da3,2725042bbe13bc67] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534171 -1 -1 , error= 2025-10-27 11:02:51.455 INFO 2458 --- [host-check-clockdiff5,f9bcdaaa9aab3da3,66e9a0fcc0868186] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534171 0 0 , error= 2025-10-27 11:02:51.459 INFO 2458 --- [host-check-clockdiff1,f9bcdaaa9aab3da3,d84fc156b44c3719] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534171 0 0 , error= 2025-10-27 11:02:51.507 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:02:51.509 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:02:51.508956687+08:00 2025-10-27 11:02:51.510 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:02:51.509+08:00 2025-10-27 11:02:51.511 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:02:51.512 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:02:51.512 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:02:51.513 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:02:51.514 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:02:51.513961143+08:00 2025-10-27 11:02:51.514 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:02:51.514 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:02:51.514+08:00 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-agent-status1,d7ea64c0537586c3,06bb314bbc2f7c12] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:02:51.515658783+08:00 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:02:51.514681695+08:00 2025-10-27 11:02:51.515 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:02:51.516 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:02:51.515+08:00 2025-10-27 11:02:51.516 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:02:51.515+08:00 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status4,d7ea64c0537586c3,7effe4aa251c6b4f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:02:51.517035932+08:00 2025-10-27 11:02:51.517 INFO 2458 --- [host-check-agent-status2,d7ea64c0537586c3,8710f27ea063376a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:02:51.518 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:02:51.517+08:00 2025-10-27 11:02:51.519 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:02:51.522 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:02:51.522 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:02:51.509+08:00 and status AVAILABLE 2025-10-27 11:02:51.522 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:02:51.522 INFO 2458 --- [host-check-agent-status3,d7ea64c0537586c3,3d9b65458f0a9a4b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:02:51.522 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:02:51.514+08:00 and status AVAILABLE 2025-10-27 11:02:51.523 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:02:51.515+08:00 and status AVAILABLE 2025-10-27 11:02:51.524 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:02:51.515+08:00 and status AVAILABLE 2025-10-27 11:02:51.525 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:02:51.526 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:02:51.517+08:00 and status AVAILABLE 2025-10-27 11:02:51.526 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:02:51.527 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:02:51.527 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:02:51.527 INFO 2458 --- [host-check-agent-status5,d7ea64c0537586c3,8defae3616a96325] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:02:51.528 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:02:51.529 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:02:51.529 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:02:51.529 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:02:51.545 INFO 2458 --- [ocp-schedules-24,2572e65252ec337e,116a954e4ba8b0a4] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:02:51.546 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:02:51.547 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:02:51.548 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:51.549 INFO 2458 --- [ocp-schedules-12,4f4140e0267031a1,e0d54e25b0c7d80c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:02:51.549 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:02:51.551 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:02:51.551 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:51.551 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:51.552 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:02:51.554 INFO 2458 --- [ocp-schedules-14,6c5cd723eced9ea6,8c41df8225001b66] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:02:51.554 INFO 2458 --- [ocp-schedules-12,4f4140e0267031a1,e0d54e25b0c7d80c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:51.554 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:02:51.555 INFO 2458 --- [ocp-schedules-12,4f4140e0267031a1,e0d54e25b0c7d80c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:02:51.557 INFO 2458 --- [ocp-schedules-24,2572e65252ec337e,116a954e4ba8b0a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:51.557 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:51.558 INFO 2458 --- [ocp-schedules-14,6c5cd723eced9ea6,8c41df8225001b66] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:51.566 INFO 2458 --- [ocp-schedules-17,131b41680f2a98f8,78251d5526f451c1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:02:51.569 INFO 2458 --- [ocp-schedules-21,0de96654da02bf0e,1f3a67329ad85e3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:02:51.570 INFO 2458 --- [ocp-schedules-13,13c42962812a3114,c48e0cd31c5e97a7] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:02:51.570 INFO 2458 --- [ocp-schedules-18,30dcb2b2a33544fa,b853d9298359cc41] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:02:51.574 INFO 2458 --- [ocp-schedules-17,131b41680f2a98f8,78251d5526f451c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:51.582 INFO 2458 --- [ocp-schedules-18,30dcb2b2a33544fa,b853d9298359cc41] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:02:51.583 INFO 2458 --- [ocp-schedules-21,0de96654da02bf0e,1f3a67329ad85e3d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:02:51.616 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.622 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.623 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:51.624 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:02:51.628 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, 2025-10-27 11:02:51.630 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:51.632 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.633 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:02:51.634 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.635 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:51.636 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.637 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:51.637 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:51.638 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.639 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:02:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:51.641 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:02:51.648 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,66d70d948e3c6006,3e3da3f27020e6ae] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=66d70d948e3c6006, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:02:51.648 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,66d70d948e3c6006,3e3da3f27020e6ae] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=66d70d948e3c6006, duration=1 ms] 2025-10-27 11:02:51.649 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.650 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:02:51.652 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:02:51.652 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:02:51.652 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.655 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:02:51.656 INFO 2458 --- [ocp-schedules-24,2572e65252ec337e,116a954e4ba8b0a4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:02:51.658 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:02:51.664 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:02:51.664 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.666 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.667 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:51.667 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:02:51.670 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.670 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.670 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.671 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.671 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:02:51.671 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.671 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:02:51.672 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.673 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:02:51.689 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,297c9020290affbd,f20c1327749be9cc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=297c9020290affbd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:02:51.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,297c9020290affbd,f20c1327749be9cc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=297c9020290affbd, duration=1 ms] 2025-10-27 11:02:51.707 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.707 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.708 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:02:51.708 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.708 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:02:51.708 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.709 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:02:51.712 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:02:51.719 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.721 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.722 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.723 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.725 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.726 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.727 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.730 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:02:51.732 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:02:51.731349+08:00, lastProcessTime=2025-10-27T11:02:41.718430+08:00 2025-10-27 11:02:51.735 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.736 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.736 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.737 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.738 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.739 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:02:51.739 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.742 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:02:51.742 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.743 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.743 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.743 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.743 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.743 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.743 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.744 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.744 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.744 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:51.745 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.745 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.746 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.747 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.748 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:02:51.749 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.749 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.749 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.751 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.752 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:02:51.752 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:02:51.753 INFO 2458 --- [ocp-schedules-17,131b41680f2a98f8,78251d5526f451c1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:02:51.758 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.769 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.772 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.775 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.776 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:51.777 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:51.777 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:51.778 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:51.780 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:02:51.780 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T10:56:51.292, end=2025-10-27T11:01:51.61} 2025-10-27 11:02:51.781 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:02:51.793 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:02:51.797 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:02:51.798 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:02:51.799 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:02:51.810 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:02:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:51.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:51.825 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.829 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:02:51.830 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:51.833 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:02:51.841 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:02:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.843 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:02:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.843 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:02:51.843 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:02:51.844 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:02:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:51.868 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:51.875 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:51.878 INFO 2458 --- [ocp-schedules-4,18e9132edee2d5d5,ccf014b1c94b59fa] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:02:51.878 INFO 2458 --- [ocp-schedules-4,18e9132edee2d5d5,ccf014b1c94b59fa] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:02:51.883 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:51.883 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:02:51.886 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:51.890 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:51.891 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:02:51.896 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:02:51.907 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:02:51.911 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:02:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:51.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:02:51.921 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:02:51.928 INFO 2458 --- [host-check-host-status4,748f69abcdd9ad27,69205519cbad35cc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:02:51.932 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:02:51.938 INFO 2458 --- [host-check-host-status3,748f69abcdd9ad27,4108e32e212fc1d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:02:51.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:02:51.946 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:02:51.948 INFO 2458 --- [host-check-host-status5,748f69abcdd9ad27,d2a84b0919771b13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:02:51.949 INFO 2458 --- [host-check-host-status2,748f69abcdd9ad27,e5c00f2be2bd5e8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:02:51.954 INFO 2458 --- [host-check-host-status1,748f69abcdd9ad27,c5c8c23582566192] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:02:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.963 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.964 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:02:51.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:51.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:51.966 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:02:51.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:52.002 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:02:52.011 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.011 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:52.028 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:02:51.731349+08:00 2025-10-27 11:02:52.046 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:02:52.046 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:02:52.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:52.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:02:52.069 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:02:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:52.092 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:52.097 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.097 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:02:52.121 INFO 2458 --- [pool-overview-container-7,b3d67baa431d11c2,78920a7aee9043c4] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.121 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:02:52.121 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:02:52.126 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.127 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:02:52.168 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.169 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:02:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:02:52.237 INFO 2458 --- [pool-overview-container-5,84f877cf2c486d9f,26532bb2444c2e39] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:52.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.255 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:02:52.260 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:02:52.275 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:02:52.287 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:02:52.290 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:02:52.290 INFO 2458 --- [pool-overview-container-6,3e6200e58ecd79a3,80538e16bb843e82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.292 INFO 2458 --- [pool-overview-container-6,3e6200e58ecd79a3,80538e16bb843e82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:02:52.307 INFO 2458 --- [pool-12-thread-16,84f877cf2c486d9f,7a24dfb26a563c77] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:02:52.310 INFO 2458 --- [pool-12-thread-13,84f877cf2c486d9f,cb7056453e7af314] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:02:52.312 INFO 2458 --- [pool-12-thread-14,84f877cf2c486d9f,757e727063bfe7a1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:02:52.313 INFO 2458 --- [pool-12-thread-15,84f877cf2c486d9f,a855edb775de1839] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:02:52.315 INFO 2458 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.316 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:02:52.316 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:02:52.317 INFO 2458 --- [ForkJoinPool-7-worker-11,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.318 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:02:52.318 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:02:52.334 INFO 2458 --- [pool-12-thread-15,84f877cf2c486d9f,a855edb775de1839] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:02:52.334 INFO 2458 --- [pool-12-thread-15,84f877cf2c486d9f,a855edb775de1839] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:02:52.335 INFO 2458 --- [pool-12-thread-13,84f877cf2c486d9f,cb7056453e7af314] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:02:52.344 INFO 2458 --- [ForkJoinPool-7-worker-4,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.345 INFO 2458 --- [ForkJoinPool-7-worker-13,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:02:52.352 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.353 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:02:52.354 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.354 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:02:52.354 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:02:52.374 INFO 2458 --- [host-check-clockdiff4,f9bcdaaa9aab3da3,8c91a3fc2db95bf6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534172 -1 -1 , error= 2025-10-27 11:02:52.375 INFO 2458 --- [pool-12-thread-13,84f877cf2c486d9f,cb7056453e7af314] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:02:52.379 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.379 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.380 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.380 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.380 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.381 INFO 2458 --- [pool-12-thread-13,84f877cf2c486d9f,cb7056453e7af314] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:02:52.381 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:02:52.384 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.385 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.385 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.385 INFO 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:02:52.390 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.392 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:02:52.403 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.403 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.403 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.403 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.404 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.406 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.406 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:02:52.406 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:02:52.407 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.408 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:02:52.408 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:02:52.414 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:02:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:02:52.415 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:02:52.416 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.417 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.417 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:02:52.417 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.417 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.418 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:02:52.419 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:02:52.432 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.433 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:02:52.435 WARN 2458 --- [ForkJoinPool-7-worker-11,84f877cf2c486d9f,55714bc85a297c17] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:02:52.436 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:02:52.440 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.441 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:02:52, 2025-10-27 11:02:52] 2025-10-27 11:02:52.451 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:02:52.452 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.452 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.452 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.452 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.452 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.453 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:02:52.454 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.454 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.454 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.455 INFO 2458 --- [ForkJoinPool-7-worker-13,84f877cf2c486d9f,59907e98347d2873] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:02:52.456 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:02:52.457 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.459 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:02:52.462 INFO 2458 --- [ocp-async-5,e50954416d72f38e,deff46efb2b7979e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:52.462 WARN 2458 --- [ocp-async-7,b2fd22d05949e0b4,87bd5cf3b97a9eb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:52.468 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:02:52.469 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.470 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:02:52, 2025-10-27 11:02:52] 2025-10-27 11:02:52.473 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:02:52.473 INFO 2458 --- [ForkJoinPool-7-worker-2,84f877cf2c486d9f,3a0c1359c5b40eb1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:02:52.474 INFO 2458 --- [pool-12-thread-16,84f877cf2c486d9f,7a24dfb26a563c77] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:02:52.482 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.482 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.482 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.482 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.483 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.486 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:02:52.487 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.488 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:02:52.488 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:02:52.489 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.490 INFO 2458 --- [ForkJoinPool-7-worker-4,84f877cf2c486d9f,02a5f137d1fb7332] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:02:52.506 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:52.506 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.506 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.507 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.507 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.507 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.508 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:02:52.514 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:52.519 INFO 2458 --- [pool-12-thread-14,84f877cf2c486d9f,757e727063bfe7a1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:02:52.523 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.528 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:02:52.563 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:02:52.563 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:02:52.563 INFO 2458 --- [ocp-schedules-14,6c5cd723eced9ea6,8c41df8225001b66] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:02:52.614 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:52.615 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.615 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.615 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:52.615 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:52.615 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:52.619 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:02:53.164 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:02:53.232 INFO 2458 --- [ocp-schedules-3,75f426016cd7f7c6,ba209b8064748a0f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:02:53.237 INFO 2458 --- [ocp-schedules-3,75f426016cd7f7c6,ba209b8064748a0f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:02:53.707 INFO 2458 --- [prometheus-minute-schedule-1,c66ffcc073db271e,5329e8f370073cfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:53.715 INFO 2458 --- [prometheus-minute-schedule-1,c66ffcc073db271e,5329e8f370073cfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:53.891 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:53.893 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:54.194 INFO 2458 --- [ocp-schedules-17,32782fc3bc1efb78,5a491a31b850d726] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:02:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:02:54.462 INFO 2458 --- [ocp-async-10,2631585ca4c39bfa,0e2095ccbbce781e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:54.462 WARN 2458 --- [ocp-async-1,711c5d181c4bede0,dec3ec154cfe380a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:54.652 INFO 2458 --- [ocp-schedules-11,49066eb0c4283360,d48ea3c580c18824] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:02:54.661 INFO 2458 --- [ocp-schedules-11,49066eb0c4283360,d48ea3c580c18824] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:02:54.745 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:02:54.746 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.749 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:02:54.760 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.761 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.786 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.787 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.796 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.796 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.797 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:02:54.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.797 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.801 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:02:54.814 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.815 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.817 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.817 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.820 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.820 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.820 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.821 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.844 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.845 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.848 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.848 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.848 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.849 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.873 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.874 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.875 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.875 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.876 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:02:54.878 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.878 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.878 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.879 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.907 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.907 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.908 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.915 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.916 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:54.917 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.917 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:54.918 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.919 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:54.920 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.921 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:54.921 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.922 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:54.923 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.924 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:54.925 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:02:54.931 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:02:54.934 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:02:54.937 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.937 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:02:54.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:54.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:54.969 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.970 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:02:54.981 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:54.982 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:54.982 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:02:54.992 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:54.992 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:02:54.992 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:54.992 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:02:54.996 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:02:54.996 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:02:55.002 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.002 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.002 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.003 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:02:55.005 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.005 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.005 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.005 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:02:55.033 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.033 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.033 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.033 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:02:55.063 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.063 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.063 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.064 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:02:55.113 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.113 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.114 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.115 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:55.115 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.115 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.115 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.116 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:55.116 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.117 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.118 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:55.118 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.118 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.119 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.119 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:55.120 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.120 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.120 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.121 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:55.121 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.121 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.121 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.122 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:55.123 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.123 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.123 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.124 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:55.124 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.124 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.124 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.125 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:55.125 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.125 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.126 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.127 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:55.127 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.127 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.127 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.128 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:02:55.129 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.129 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.129 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.130 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:02:55.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.130 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.132 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:02:55.133 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:02:55.136 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:02:55.226 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:02:55.226 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.226 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.226 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:02:55.226 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:02:55.227 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:02:55.227 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:02:55.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:02:55.258 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:02:55.259 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:02:55.259 INFO 2458 --- [ocp-schedules-11,49066eb0c4283360,d48ea3c580c18824] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:02:55.624 INFO 2458 --- [ocp-schedules-1,abd3b91fab6a01d0,aeafcb93a33bf05d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:02:55.703 INFO 2458 --- [ocp-schedules-8,ade59f049f2754e5,13323f404e2a2eef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:02:55.735 WARN 2458 --- [scanner-112120797-1,ade59f049f2754e5,473800d0d2b468d2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:02:55.816 INFO 2458 --- [alarm-detect-8,ade59f049f2754e5,3d941b0aed30281d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534150 ~ 1761534165 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534165, value=67.357)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534165, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534165=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:02:56.462 WARN 2458 --- [ocp-async-2,efb38b098782d9d5,5d1af94de1f7463a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:56.462 INFO 2458 --- [ocp-async-2,2781c8b70dcd1f14,5e07b52677b9096a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:56.707 INFO 2458 --- [prometheus-minute-schedule-1,8872401db78bbcde,556e712d4f4daecf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:56.715 INFO 2458 --- [prometheus-minute-schedule-1,8872401db78bbcde,556e712d4f4daecf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:56.893 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:56.893 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:02:58.462 WARN 2458 --- [ocp-async-9,6275da695e27ae61,f93a020025215a35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:02:58.462 INFO 2458 --- [ocp-async-7,8b8f6ad04d6958ab,ecc2e207e4d24094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:02:58.505 INFO 2458 --- [ocp-async-5,bbf7a20a3e0f4fc0,867d77a18c70ff08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:02:59.080 INFO 2458 --- [ocp-schedules-2,ad63ac8726cee50a,425918f752b6cd51] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:02:59.707 INFO 2458 --- [prometheus-minute-schedule-1,9c374f06433c5bdf,fe6679fa566ec568] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534180, interval=60 2025-10-27 11:02:59.716 INFO 2458 --- [prometheus-minute-schedule-1,9c374f06433c5bdf,fe6679fa566ec568] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:02:59.894 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:02:59.895 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:00.462 WARN 2458 --- [ocp-async-1,c9258ef5eeea4a56,2b454c31b4acec1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:00.462 INFO 2458 --- [ocp-async-4,fb9e01ae2eafad69,fe1035759a41e61a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:00.536 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,68ca46f7a1ea9b29,ec2e21bac46023cd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=68ca46f7a1ea9b29, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:03:00.536 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,68ca46f7a1ea9b29,ec2e21bac46023cd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=68ca46f7a1ea9b29, duration=1 ms] 2025-10-27 11:03:01.702 INFO 2458 --- [ocp-schedules-21,2c0541efc22a934e,3aa8fd2eaa7f6cb3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:03:01.711 INFO 2458 --- [ocp-schedules-21,2c0541efc22a934e,3aa8fd2eaa7f6cb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:01.715026+08:00, lastProcessTime=2025-10-27T11:02:51.731349+08:00 2025-10-27 11:03:01.754 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:01.755 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:03:01.772 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:01.772 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:01.772 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:01.772 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:01.772 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:01.773 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:03:01.786 INFO 2458 --- [ocp-schedules-21,2c0541efc22a934e,3aa8fd2eaa7f6cb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:03:01.934 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:01.715026+08:00 2025-10-27 11:03:02.463 INFO 2458 --- [ocp-async-8,e61eb7695e2c5d08,dfc0ba7f12435dc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:02.463 WARN 2458 --- [ocp-async-2,4d0fd0f530c76110,69a75b1bac95c914] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:02.707 INFO 2458 --- [prometheus-minute-schedule-1,3094267fa0bacd29,7beab45980287b0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:02.714 INFO 2458 --- [prometheus-minute-schedule-1,3094267fa0bacd29,7beab45980287b0d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:02.895 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:02.896 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:03.248 INFO 2458 --- [ocp-schedules-20,f48ecb77fa8bdded,c98885ae1d924c91] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:03.252 INFO 2458 --- [ocp-schedules-20,f48ecb77fa8bdded,c98885ae1d924c91] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:04.463 WARN 2458 --- [ocp-async-5,80ca26ea6fffb7e0,59c12641b7b629ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:04.463 INFO 2458 --- [ocp-async-7,2c4d3a4a514d2293,0287f5a715ccf102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:05.631 INFO 2458 --- [ocp-schedules-28,f50ce796b0473aba,3651105b7eb2c54d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:05.707 INFO 2458 --- [prometheus-minute-schedule-1,9914589693c9116d,de3aa09a8d3e1bee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:05.714 INFO 2458 --- [prometheus-minute-schedule-1,9914589693c9116d,de3aa09a8d3e1bee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:05.822 INFO 2458 --- [ocp-schedules-5,2c062b7961b4a85b,9ba5db75afa7ac2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:05.896 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:05.897 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:05.927 WARN 2458 --- [scanner-112120980-1,2c062b7961b4a85b,1cebe22bff43e895] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:06.111 INFO 2458 --- [metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=797, affectRows=797 2025-10-27 11:03:06.111 INFO 2458 --- [metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=797, validCount=797 2025-10-27 11:03:06.463 INFO 2458 --- [ocp-async-1,9d8cc2958ef1fdc2,fd267e601d2d13b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:06.463 WARN 2458 --- [ocp-async-10,c715e244fea75c34,0dd6245a6ebe74ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:06.537 INFO 2458 --- [ocp-schedules-4,dadc58a4a616dd2a,71469f625ade36a9] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:03:06.546 INFO 2458 --- [ocp-schedules-4,dadc58a4a616dd2a,71469f625ade36a9] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:06.546 INFO 2458 --- [ocp-schedules-4,dadc58a4a616dd2a,71469f625ade36a9] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:06.915 INFO 2458 --- [metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=882, affectRows=882 2025-10-27 11:03:06.915 INFO 2458 --- [metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=882, validCount=882 2025-10-27 11:03:07.353 INFO 2458 --- [metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2025-10-27 11:03:07.353 INFO 2458 --- [metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=1000, validCount=1000 2025-10-27 11:03:08.463 INFO 2458 --- [ocp-async-2,6df1472190a818c6,62fd3c6a0e8c7139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:08.463 WARN 2458 --- [ocp-async-6,2aad61d9a8665a0d,b2ad10c15a70e23a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:08.505 INFO 2458 --- [ocp-async-9,648861d5dfa5660e,cabbf22f5e595db4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:08.707 INFO 2458 --- [prometheus-minute-schedule-1,5cf860771a13c81e,c676f3d7dde5f637] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:08.717 INFO 2458 --- [prometheus-minute-schedule-1,5cf860771a13c81e,c676f3d7dde5f637] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:08.897 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:08.899 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:09.002 INFO 2458 --- [metric-archive-minute-5,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2025-10-27 11:03:09.002 INFO 2458 --- [metric-archive-minute-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=1000, validCount=1000 2025-10-27 11:03:09.013 INFO 2458 --- [metric-archive-minute-6,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2025-10-27 11:03:09.014 INFO 2458 --- [metric-archive-minute-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=1000, validCount=1000 2025-10-27 11:03:09.075 INFO 2458 --- [metric-archive-minute-2,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2025-10-27 11:03:09.075 INFO 2458 --- [metric-archive-minute-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=1000, validCount=1000 2025-10-27 11:03:10.013 INFO 2458 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:03:10.013 INFO 2458 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:03:10.052 INFO 2458 --- [Collect obproxy1,16e5990023470540,d8d1cbec8bf580a5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:03:10.052 INFO 2458 --- [Check obproxy1,b4b10d64fb830fdd,e8681832b79e80ee] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:03:10.052 INFO 2458 --- [Collect obproxy1,16e5990023470540,d8d1cbec8bf580a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:03:10.053 INFO 2458 --- [Check obproxy1,b4b10d64fb830fdd,e8681832b79e80ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:03:10.080 INFO 2458 --- [Collect obproxy1,16e5990023470540,d8d1cbec8bf580a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:03:10.083 INFO 2458 --- [Check obproxy1,b4b10d64fb830fdd,e8681832b79e80ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:03:10.084 INFO 2458 --- [Collect obproxy1,16e5990023470540,d8d1cbec8bf580a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:03:10.086 INFO 2458 --- [Check obproxy1,b4b10d64fb830fdd,e8681832b79e80ee] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:03:10.086 INFO 2458 --- [Check obproxy1,b4b10d64fb830fdd,e8681832b79e80ee] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:03:10.087 INFO 2458 --- [Collect obproxy1,16e5990023470540,d8d1cbec8bf580a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:03:10.107 INFO 2458 --- [metric-archive-minute-1,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=1000, affectRows=1000 2025-10-27 11:03:10.107 INFO 2458 --- [metric-archive-minute-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=1000, validCount=1000 2025-10-27 11:03:10.185 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:03:10.270 INFO 2458 --- [metric-archive-minute-8,,] c.o.ocp.monitor.store.MetricDataStore : storeMetricHourDataEntity, MetricHourDataEntity=854, affectRows=854 2025-10-27 11:03:10.270 INFO 2458 --- [metric-archive-minute-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup minute to hour, timestamp=1761530400, nodeCount=854, validCount=854 2025-10-27 11:03:10.463 WARN 2458 --- [ocp-async-7,2d82d46014d262fc,8eb83c8c45ca0082] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:10.463 INFO 2458 --- [ocp-async-9,425b28c325690424,c2eba508d5bc8da6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:11.707 INFO 2458 --- [prometheus-minute-schedule-1,4dae64689ef74b35,5f9ac3d13e5ccd92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:11.714 INFO 2458 --- [prometheus-minute-schedule-1,4dae64689ef74b35,5f9ac3d13e5ccd92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:11.714127+08:00, lastProcessTime=2025-10-27T11:03:01.715026+08:00 2025-10-27 11:03:11.899 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:11.900 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:11.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:11.714127+08:00 2025-10-27 11:03:12.463 WARN 2458 --- [ocp-async-10,c8e625a869bf4aac,dc5817395fc0b452] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:12.463 INFO 2458 --- [ocp-async-1,392ed0ce07a117f3,6d5d4bbe4355ac68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:13.261 INFO 2458 --- [ocp-schedules-18,8415a14a9a8e7dec,dcf28b752e0b74a8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:13.264 INFO 2458 --- [ocp-schedules-18,8415a14a9a8e7dec,dcf28b752e0b74a8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:13.372 INFO 2458 --- [ocp-schedules-26,c7622a8b1d83ea51,92c71470f34688cb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:03:13.372 INFO 2458 --- [ocp-schedules-26,c7622a8b1d83ea51,92c71470f34688cb] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:03:13.379 INFO 2458 --- [ocp-schedules-26,c7622a8b1d83ea51,92c71470f34688cb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:03:13.379 INFO 2458 --- [ocp-schedules-26,c7622a8b1d83ea51,92c71470f34688cb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:03:13.684 INFO 2458 --- [ocp-schedules-9,4a75afad68704248,26beaed543dc10dc] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:03:13.684 INFO 2458 --- [ocp-schedules-9,4a75afad68704248,26beaed543dc10dc] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:03:13.743 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:03:13.743 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:03:13.744 INFO 2458 --- [Collect all tenants major compactions5,4a75afad68704248,147181f232402142] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:03:13.744 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:03:13.744 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:03:13.744 INFO 2458 --- [Collect all tenants major compactions6,4a75afad68704248,cdaf8b0c58407265] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:03:13.780 INFO 2458 --- [Collect all tenants major compactions6,4a75afad68704248,cdaf8b0c58407265] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.780 INFO 2458 --- [Collect all tenants major compactions5,4a75afad68704248,147181f232402142] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.781 INFO 2458 --- [Collect all tenants major compactions5,4a75afad68704248,147181f232402142] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:03:13.782 INFO 2458 --- [Collect all tenants major compactions6,4a75afad68704248,cdaf8b0c58407265] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:03:13.792 INFO 2458 --- [Collect all tenants major compactions6,4a75afad68704248,cdaf8b0c58407265] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:03:13.793 INFO 2458 --- [Collect all tenants major compactions5,4a75afad68704248,147181f232402142] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:03:13.796 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.796 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.796 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.796 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.797 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.798 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:03:13.800 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.800 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.800 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.800 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.800 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.801 INFO 2458 --- [Collect all tenants major compactions1,4a75afad68704248,6e0c69cd15b420dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.802 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.803 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.803 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.803 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:13.803 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:13.803 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:13.805 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:03:13.805 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:03:13.806 INFO 2458 --- [Collect all tenants major compactions4,4a75afad68704248,919a560c06d25a67] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:03:13.808 INFO 2458 --- [Collect all tenants major compactions3,4a75afad68704248,d76f62eea8e41475] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:03:13.808 INFO 2458 --- [Collect all tenants major compactions2,4a75afad68704248,f65a7914ab1a96f7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:03:13.809 INFO 2458 --- [ocp-schedules-9,4a75afad68704248,26beaed543dc10dc] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:03:14.022 INFO 2458 --- [ocp-schedules-12,cd0b02282d389d28,d1b56e159261433a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:03:14.022+08:00, clusters=[1, 2] 2025-10-27 11:03:14.026 INFO 2458 --- [ocp-schedules-12,cd0b02282d389d28,d1b56e159261433a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:03:14.026 INFO 2458 --- [ocp-schedules-12,cd0b02282d389d28,d1b56e159261433a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:03:14.463 WARN 2458 --- [ocp-async-8,2df391cb8cb0629a,9d525df23552a915] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:14.463 INFO 2458 --- [ocp-async-5,57aaaf1deaee1e73,544e3f1f4914f727] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:14.707 INFO 2458 --- [prometheus-minute-schedule-1,2889c3b286770543,6c04e41067824382] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:14.715 INFO 2458 --- [prometheus-minute-schedule-1,2889c3b286770543,6c04e41067824382] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:14.900 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:14.901 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:15.639 INFO 2458 --- [ocp-schedules-12,74b3d1ced5b308e2,8fe72229976a2c0c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:16.082 INFO 2458 --- [ocp-schedules-17,21dc4612aa042bae,194300010bcd09fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:16.275 WARN 2458 --- [scanner-112122763-1,21dc4612aa042bae,8c5c4c1bace0a47e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:16.326 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534120, nodeCount=1000, affectRows=1000 2025-10-27 11:03:16.331 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534120, nodeCount=994, validCount=992 2025-10-27 11:03:16.463 WARN 2458 --- [ocp-async-5,5c9c5a4ed1ecb386,3226c68cc14dd733] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:16.464 INFO 2458 --- [ocp-async-2,53b914a4c9ded1c4,c07e7e784f6691e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:16.535 INFO 2458 --- [alarm-detect-13,21dc4612aa042bae,7485cd3eee1f804e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534165 ~ 1761534186 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534180, value=65.973)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534180, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534180=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:03:16.538 INFO 2458 --- [alarm-detect-13,21dc4612aa042bae,7485cd3eee1f804e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:03:16.745 ERROR 2458 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538020) Duplicate entry '835014129065861120-1761534060' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538020) Duplicate entry '835014129065861120-1761534060' for key 'PRIMARY' 2025-10-27 11:03:17.707 INFO 2458 --- [prometheus-minute-schedule-1,8647ef507ce915a9,deabe322594c023d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:17.715 INFO 2458 --- [prometheus-minute-schedule-1,8647ef507ce915a9,deabe322594c023d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:17.901 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:17.902 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:18.209 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534120, nodeCount=74, affectRows=74 2025-10-27 11:03:18.210 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534120, nodeCount=73, validCount=73 2025-10-27 11:03:18.464 WARN 2458 --- [ocp-async-9,f034c44544b359e6,f765c115bfa32ae0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:18.464 INFO 2458 --- [ocp-async-9,840a5f6cd41a9da9,00ebbab30169cfa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:18.505 INFO 2458 --- [ocp-async-1,3da8d7a99c40093f,cc123034787185ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:18.912 INFO 2458 --- [ocp-schedules-3,769f877cc53de29d,527e425b34e30127] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:03:20.464 WARN 2458 --- [ocp-async-1,0f39c27fa81b6947,16a8e582b8967659] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:20.464 INFO 2458 --- [ocp-async-6,31ebbbdfd741bc8a,083a3e15aa712de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:20.492 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534120, nodeCount=420, affectRows=420 2025-10-27 11:03:20.496 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534120, nodeCount=1000, affectRows=1000 2025-10-27 11:03:20.500 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534120, nodeCount=998, validCount=994 2025-10-27 11:03:20.521 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534120, nodeCount=401, affectRows=401 2025-10-27 11:03:20.707 INFO 2458 --- [prometheus-minute-schedule-1,300912f8b3cb34ea,2f7cd627beff6976] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:20.715 INFO 2458 --- [prometheus-minute-schedule-1,300912f8b3cb34ea,2f7cd627beff6976] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:20.903 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:20.904 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:21.410 INFO 2458 --- [ocp-schedules-9,382d371462456e0b,9edce58a82def140] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:21.410 INFO 2458 --- [ocp-schedules-19,cee3291c58c14557,a25700553a3e765d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:21.410 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:03:21.410 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:03:21.411 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:03:21.411 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:03:21.411 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:03:21.412 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:03:21.412 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:03:21.412 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:03:21.413 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:03:21.413 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:03:21.425 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:21.426 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:21.426 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:21.426 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:21.426 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:21.427 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:21.430 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:21.430 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:21.430 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:21.430 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:21.431 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:21.431 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:21.431 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:21.432 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:21.432 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:21.432 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:21.433 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:21.433 INFO 2458 --- [ocp-schedules-11,7b93e779b790c655,0f038b8a07b5bb4e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:21.433 INFO 2458 --- [host-check-clockdiff1,7b93e779b790c655,f80d7daab8143c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:03:21.433 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:21.434 INFO 2458 --- [host-check-clockdiff2,7b93e779b790c655,4a4e8d8356412248] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:03:21.434 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:21.435 INFO 2458 --- [host-check-clockdiff3,7b93e779b790c655,0ce505617cdec0da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:03:21.436 INFO 2458 --- [host-check-clockdiff5,7b93e779b790c655,1cdb799940f05bad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:03:21.436 INFO 2458 --- [host-check-clockdiff4,7b93e779b790c655,a2bb0ca9655bc9e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:03:21.454 INFO 2458 --- [host-check-clockdiff2,7b93e779b790c655,4a4e8d8356412248] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534201 -1 -1 , error= 2025-10-27 11:03:21.454 INFO 2458 --- [ocp-schedules-4,30269ff70af0e286,2133f6d5fd602472] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:03:21.457 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:21.468 INFO 2458 --- [host-check-clockdiff4,7b93e779b790c655,a2bb0ca9655bc9e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534201 0 0 , error= 2025-10-27 11:03:21.486 INFO 2458 --- [host-check-clockdiff1,7b93e779b790c655,f80d7daab8143c83] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534201 0 0 , error= 2025-10-27 11:03:21.526 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:03:21.529 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:03:21.529 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:03:21.530 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:03:21.531 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:03:21.530944841+08:00 2025-10-27 11:03:21.532 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:03:21.532 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:03:21.531+08:00 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:03:21.533590031+08:00 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status3,382d371462456e0b,fcf3339224d8e469] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:03:21.534+08:00 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:03:21.534 INFO 2458 --- [host-check-agent-status5,382d371462456e0b,0adf042004f434aa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:03:21.537 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:03:21.537 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:03:21.538 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:03:21.538 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:03:21.537525100+08:00 2025-10-27 11:03:21.538 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:03:21.538809595+08:00 2025-10-27 11:03:21.538 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:03:21.538+08:00 2025-10-27 11:03:21.539 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:03:21.538+08:00 2025-10-27 11:03:21.539 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:03:21.538412647+08:00 2025-10-27 11:03:21.540 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:03:21.539+08:00 2025-10-27 11:03:21.541 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:03:21.543 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:03:21.534+08:00 and status AVAILABLE 2025-10-27 11:03:21.543 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status1,382d371462456e0b,03f0e8ddc39ddb03] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:03:21.544 INFO 2458 --- [host-check-agent-status2,382d371462456e0b,92f1f9862762e7ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:03:21.545 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:03:21.531+08:00 and status AVAILABLE 2025-10-27 11:03:21.545 INFO 2458 --- [ocp-schedules-2,9124d3dcba04ef86,4d61f6e8f2ef077c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:03:21.546 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:03:21.546 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:03:21.546 INFO 2458 --- [host-check-agent-status4,382d371462456e0b,981aca42e06cf2e6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:03:21.547 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:03:21.548 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:03:21.538+08:00 and status AVAILABLE 2025-10-27 11:03:21.549 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:03:21.538+08:00 and status AVAILABLE 2025-10-27 11:03:21.549 INFO 2458 --- [ocp-schedules-3,6779bb0079af9f98,6c04f3249854370f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:03:21.549 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:03:21.539+08:00 and status AVAILABLE 2025-10-27 11:03:21.550 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:03:21.551 INFO 2458 --- [ocp-schedules-2,9124d3dcba04ef86,4d61f6e8f2ef077c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:21.553 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:03:21.553 INFO 2458 --- [ocp-schedules-3,6779bb0079af9f98,6c04f3249854370f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:21.553 INFO 2458 --- [ocp-schedules-3,6779bb0079af9f98,6c04f3249854370f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:03:21.554 INFO 2458 --- [ocp-schedules-10,88dcfea894ea744f,6ee149b61509bdce] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:03:21.555 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:03:21.555 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:03:21.562 INFO 2458 --- [ocp-schedules-10,88dcfea894ea744f,6ee149b61509bdce] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:21.568 INFO 2458 --- [ocp-schedules-27,3b37a31c678fee20,a59936e8bfd1e404] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:03:21.576 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:21.579 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:21.580 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:21.582 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:21.583 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:21.585 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:21.585 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:21.587 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:21.587 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:21.590 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:03:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:21.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:03:21.622 INFO 2458 --- [ocp-schedules-2,9124d3dcba04ef86,4d61f6e8f2ef077c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:03:21.631 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:21.633 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:21.635 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:21.637 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:21.641 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:21.649 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:03:21.649 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:21.654 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:03:21.655 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:03:21.668 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:03:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.673 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.673 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:21.673 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:03:21.681 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:21.682 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.682 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:03:21.683 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:03:21.685 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:03:21.690 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:03:21.703 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:03:21.707 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:03:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:21.714936+08:00, lastProcessTime=2025-10-27T11:03:11.714127+08:00 2025-10-27 11:03:21.716 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:21.717 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:21.726 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:21.732 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:03:21.733 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:03:21.739 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:21.741 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:21.742 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.743 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:21.743 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:21.744 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.747 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:03:21.748 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:21.750 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:03:21.761 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.763 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:03:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.764 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:03:21.765 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:03:21.771 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:03:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:21.785 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:03:21.794 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:21.797 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:03:21.798 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:21.801 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:03:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:21.809 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:03:21.814 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:03:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:21.829 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.829 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:21.829 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:21.829 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:21.830 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:03:21.833 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:21.843 INFO 2458 --- [host-check-host-status4,cee3291c58c14557,3648e3028d7fd361] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:03:21.843 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:03:21.847 INFO 2458 --- [host-check-host-status5,cee3291c58c14557,5ce04a5065db38fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:03:21.849 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:03:21.850 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:03:21.856 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:21.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:21.859 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:21.859 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.859 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.860 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.860 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.860 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.861 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:21.862 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:21.862 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:21.862 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:03:21.867 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:21.872 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:03:21.881 INFO 2458 --- [host-check-host-status2,cee3291c58c14557,0131a95eae5c2938] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:03:21.886 INFO 2458 --- [host-check-host-status3,cee3291c58c14557,3dd41c0d6443843e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:03:21.899 INFO 2458 --- [host-check-host-status1,cee3291c58c14557,b8862d5b1fed5875] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:03:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:21.922 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:03:21.959 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:21.960 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:21.966 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:03:21.967 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.968 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:03:21.969 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:21.970 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:03:21.985 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:21.714936+08:00 2025-10-27 11:03:21.986 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:03:21.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:21.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:22.008 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:22.009 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:22.010 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:03:22.032 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:03:22.032 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:03:22.034 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:22.035 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:22.102 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.145 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:22.145 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:22.150 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:03:22.162 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:03:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:22.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:22.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:03:22.279 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:03:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:22.280 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:03:22.282 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.282 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.282 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:03:22.317 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:03:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:03:22.379 INFO 2458 --- [host-check-clockdiff3,7b93e779b790c655,0ce505617cdec0da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534202 0 0 , error= 2025-10-27 11:03:22.391 INFO 2458 --- [host-check-clockdiff5,7b93e779b790c655,1cdb799940f05bad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534202 0 0 , error= 2025-10-27 11:03:22.426 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:03:22.426 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:03:22.426 INFO 2458 --- [ocp-schedules-10,88dcfea894ea744f,6ee149b61509bdce] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:03:22.464 WARN 2458 --- [ocp-async-5,70bd3fef2c36bd87,9125f637b03592e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:22.464 INFO 2458 --- [ocp-async-5,ae8d3ad29704a35b,ca394e5de762305b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:23.272 INFO 2458 --- [ocp-schedules-18,3949f3e49cd47648,7b95fd5efdc576a7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:23.276 INFO 2458 --- [ocp-schedules-18,3949f3e49cd47648,7b95fd5efdc576a7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:23.707 INFO 2458 --- [prometheus-minute-schedule-1,bebaa155477e56fe,23e59677a0b00da4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:23.715 INFO 2458 --- [prometheus-minute-schedule-1,bebaa155477e56fe,23e59677a0b00da4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:23.904 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:23.905 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:24.464 WARN 2458 --- [ocp-async-2,17df33c28455fcc9,786c4ceeea3ca9b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:24.464 INFO 2458 --- [ocp-async-9,f9535e22b1a59baf,c37feb4c9cb1bc29] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:24.717 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; uncategorized SQLException for SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [4012]; (conn=134538019) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again.; nested exception is java.sql.BatchUpdateException: (conn=134538019) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. 2025-10-27 11:03:24.733 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534120, nodeCount=1000, affectRows=1000 2025-10-27 11:03:24.737 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534120, nodeCount=998, validCount=994 2025-10-27 11:03:25.646 INFO 2458 --- [ocp-schedules-18,a818ecde4b0bc0b0,bb9697ca4bf1128a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:26.464 WARN 2458 --- [ocp-async-1,79749871487154a4,40c041f3d49bd194] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:26.464 INFO 2458 --- [ocp-async-8,7b5f24e431124c75,ef090d780dcd78cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:26.548 INFO 2458 --- [ocp-schedules-16,18489b33b51fbca4,3ccc80e9594c9129] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:03:26.556 INFO 2458 --- [ocp-schedules-16,18489b33b51fbca4,3ccc80e9594c9129] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:26.556 INFO 2458 --- [ocp-schedules-16,18489b33b51fbca4,3ccc80e9594c9129] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:26.573 INFO 2458 --- [ocp-schedules-26,d429e4becdab5ea1,091a664005e01bcd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:26.619 WARN 2458 --- [scanner-112122946-1,d429e4becdab5ea1,ad920c97105b0544] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:26.655 ERROR 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Archive second nodes failed. org.springframework.jdbc.UncategorizedSQLException: PreparedStatementCallback; uncategorized SQLException for SQL [REPLACE INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; SQL state [HY000]; error code [4012]; (conn=134538020) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again.; nested exception is java.sql.BatchUpdateException: (conn=134538020) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1576) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:667) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:691) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1034) at org.springframework.jdbc.core.JdbcTemplate.batchUpdate(JdbcTemplate.java:1088) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDao.write(RollupMetricDataDao.java:74) at com.oceanbase.ocp.monitor.storage.RollupMetricDataDaoProxy.write(RollupMetricDataDaoProxy.java:55) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.lambda$archiveSecond$1(OcpMetricArchiveService.java:161) at com.oceanbase.ocp.common.meter.MeterUtils.time(MeterUtils.java:66) at com.oceanbase.ocp.common.meter.DistributionSummaryGroup.time(DistributionSummaryGroup.java:184) at com.oceanbase.ocp.monitor.service.OcpMetricArchiveService.archiveSecond(OcpMetricArchiveService.java:152) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.BatchUpdateException: (conn=134538020) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:450) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:453) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:346) at com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:127) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) at org.springframework.jdbc.core.JdbcTemplate.lambda$batchUpdate$4(JdbcTemplate.java:1048) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) ... 16 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=134538020) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeBatchExceptionEpilogue(OceanBaseStatement.java:446) ... 22 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1252) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchClient(AbstractQueryProtocol.java:704) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternalBatch(JDBC4PreparedStatement.java:461) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeBatch(JDBC4PreparedStatement.java:332) ... 20 common frames omitted Caused by: java.sql.SQLException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2119) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeBatchRewrite(AbstractQueryProtocol.java:1244) ... 23 common frames omitted 2025-10-27 11:03:26.707 INFO 2458 --- [prometheus-minute-schedule-1,26be98d3113897ed,b6dc6706aec51f9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:26.718 INFO 2458 --- [prometheus-minute-schedule-1,26be98d3113897ed,b6dc6706aec51f9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:26.777 INFO 2458 --- [alarm-detect-9,d429e4becdab5ea1,b3de5dd7800b4572] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534180 ~ 1761534196 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534195, value=74.948)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534195, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534195=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:03:26.905 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:26.906 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:28.464 WARN 2458 --- [ocp-async-6,4ab76c6bd4737242,e753dbc976a79cbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:28.464 INFO 2458 --- [ocp-async-3,540d82d72a34248a,4e0cb25087d54756] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:28.503 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534120, nodeCount=1000, affectRows=1000 2025-10-27 11:03:28.506 INFO 2458 --- [ocp-async-5,076e2a34641b2289,4751d146e1c97adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:28.506 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534120, nodeCount=999, validCount=995 2025-10-27 11:03:28.790 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:03:28.790 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:03:28.802 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:03:28.802 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:03:28.812 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:03:28.812 INFO 2458 --- [ocp-schedules-5,7a01b9d5734a9682,0df0a8f510dea9b8] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:03:29.281 INFO 2458 --- [ocp-schedules-3,aeac552df0a8b6d4,23e7482f4ccec955] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:03:29.287 INFO 2458 --- [ocp-schedules-3,aeac552df0a8b6d4,23e7482f4ccec955] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:03:29.287 INFO 2458 --- [ocp-schedules-3,aeac552df0a8b6d4,23e7482f4ccec955] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:03:29.707 INFO 2458 --- [prometheus-minute-schedule-1,804e079e61a98f1e,63bfce4508072740] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:29.715 INFO 2458 --- [prometheus-minute-schedule-1,804e079e61a98f1e,63bfce4508072740] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:29.906 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:29.907 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:30.464 WARN 2458 --- [ocp-async-5,b32aa29d25645077,8cc9af137fc3ccfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:30.464 INFO 2458 --- [ocp-async-5,a02c878bbfd5665b,51f56cc1a31b7f6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:31.714276+08:00, lastProcessTime=2025-10-27T11:03:21.714936+08:00 2025-10-27 11:03:31.949 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:31.714276+08:00 2025-10-27 11:03:32.465 WARN 2458 --- [ocp-async-4,fa44e086385807da,b1a6c87798319bab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:32.465 INFO 2458 --- [ocp-async-4,f68f5ced95c56497,71fdf1451900a41e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:32.707 INFO 2458 --- [prometheus-minute-schedule-1,f4d2be858fcb02e4,4534947aec8f5ab9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:32.715 INFO 2458 --- [prometheus-minute-schedule-1,f4d2be858fcb02e4,4534947aec8f5ab9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:32.907 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:32.909 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:33.284 INFO 2458 --- [ocp-schedules-25,02cdadb21655d82f,ef056af8c7be660b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:33.288 INFO 2458 --- [ocp-schedules-25,02cdadb21655d82f,ef056af8c7be660b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:33.699 INFO 2458 --- [ocp-schedules-19,f2c934e15690590b,c3d7fdce12477201] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:03:33.699 INFO 2458 --- [ocp-schedules-19,f2c934e15690590b,c3d7fdce12477201] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:03:33.869 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:03:33.869 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:03:33.887 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:03:33.887 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:03:33.887 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:03:33.899 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:03:33.899 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:03:33.899 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:03:33.911 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:03:33.911 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:03:33.911 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:03:33.921 INFO 2458 --- [ocp-schedules-29,6124ba700fc582f1,6f2a7606821fd90d] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:03:34.113 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:03:34.113 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:03:34.182 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:34.182 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:34.225 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:34.225 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:34.225 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:34.226 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:03:34.238 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:34.238 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:34.238 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:34.239 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:03:34.297 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:34.298 INFO 2458 --- [ocp-schedules-8,fe12e729eaa0940f,ae93ca1f75803ba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:03:34.465 WARN 2458 --- [ocp-async-7,34cf9753088ef347,150901dcd821f500] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:34.465 INFO 2458 --- [ocp-async-7,9952fe120bca5123,4bc968c20fcd6ecd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:35.654 INFO 2458 --- [ocp-schedules-1,b6d4d1c9bfdbc53a,a9976cd16fa2c6e6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:35.707 INFO 2458 --- [prometheus-minute-schedule-1,a731f6c2584ec2a8,41d5e3367aad9a3f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:35.714 INFO 2458 --- [prometheus-minute-schedule-1,a731f6c2584ec2a8,41d5e3367aad9a3f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:35.909 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:35.910 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:36.465 WARN 2458 --- [ocp-async-9,18a82cd720f154b5,e6eb8f791ff8bf64] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:36.465 INFO 2458 --- [ocp-async-2,41b0cd56ba6c49ef,3e39905dbd08ba01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:36.546 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:36.546 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:03:36.546 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:03:36.546 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:03:36.546 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:03:36.546 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:03:36.558 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:36.558 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:36.559 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:36.559 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:36.559 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:36.562 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:36.563 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:36.563 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:36.563 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:36.564 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:36.619 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:03:36.622 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:03:36.624 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:03:36.624 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:03:36.626 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:03:36.628 INFO 2458 --- [ocp-schedules-30,90b1300759bd048a,1a2a050755a6a247] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:03:36.782 INFO 2458 --- [ocp-schedules-28,8d50f47b02263dec,e1296e90fd3e4b41] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:36.817 WARN 2458 --- [scanner-112123200-1,8d50f47b02263dec,7c74abc85eeca10a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:37.708 INFO 2458 --- [prometheus-cache-monitor1,dbaba0fc4398dc59,b93ca0bba019d22b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20297,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5075,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7860} 2025-10-27 11:03:37.708 INFO 2458 --- [prometheus-cache-monitor1,dbaba0fc4398dc59,b93ca0bba019d22b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:03:38.467 WARN 2458 --- [ocp-async-1,2e0badc638f85115,f767a63c9abdd8fb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:38.467 INFO 2458 --- [ocp-async-8,56cf3c9f7af1e9a3,0850a11bd39eae2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:38.506 INFO 2458 --- [ocp-async-4,5ab03daaeb793711,1d43d2ae499b020a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:38.557 INFO 2458 --- [ocp-schedules-10,d5851502dc81a2c4,d12021e84a21732f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:03:38.564 INFO 2458 --- [ocp-schedules-10,d5851502dc81a2c4,d12021e84a21732f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:38.564 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:03:38.564 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:03:38.565 INFO 2458 --- [ocp-schedules-10,d5851502dc81a2c4,d12021e84a21732f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:03:38.707 INFO 2458 --- [prometheus-minute-schedule-1,402b4dfa31e380c0,f2032f579c266b1d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:38.714 INFO 2458 --- [prometheus-minute-schedule-1,402b4dfa31e380c0,f2032f579c266b1d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:38.910 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:38.911 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:40.008 INFO 2458 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:03:40.009 INFO 2458 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:03:40.043 INFO 2458 --- [Check obproxy1,7f257288c9e3fd01,db7991edef73f593] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:03:40.044 INFO 2458 --- [Check obproxy1,7f257288c9e3fd01,db7991edef73f593] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:03:40.048 INFO 2458 --- [Collect obproxy1,b012481b04128f60,5e486e18890c1c65] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:03:40.048 INFO 2458 --- [Collect obproxy1,b012481b04128f60,5e486e18890c1c65] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:03:40.078 INFO 2458 --- [Check obproxy1,7f257288c9e3fd01,db7991edef73f593] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:03:40.081 INFO 2458 --- [Check obproxy1,7f257288c9e3fd01,db7991edef73f593] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:03:40.081 INFO 2458 --- [Check obproxy1,7f257288c9e3fd01,db7991edef73f593] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:03:40.082 INFO 2458 --- [Collect obproxy1,b012481b04128f60,5e486e18890c1c65] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:03:40.085 INFO 2458 --- [Collect obproxy1,b012481b04128f60,5e486e18890c1c65] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:03:40.087 INFO 2458 --- [Collect obproxy1,b012481b04128f60,5e486e18890c1c65] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:03:40.143 INFO 2458 --- [ocp-schedules-2,5eea002c627aaa7f,d07ba9f7cdde8e43] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:03:40.143 INFO 2458 --- [ocp-schedules-2,5eea002c627aaa7f,d07ba9f7cdde8e43] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:03:40.467 WARN 2458 --- [ocp-async-7,047f3cc16a9cc0b2,8b6fb727cee231a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:40.467 INFO 2458 --- [ocp-async-6,e1362794eebb7a68,dca53ce0a13b2cdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:41.707 INFO 2458 --- [prometheus-minute-schedule-1,28b7c6a4b54e88dd,c8a2d9b421d20e9e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:41.714534+08:00, lastProcessTime=2025-10-27T11:03:31.714276+08:00 2025-10-27 11:03:41.716 INFO 2458 --- [prometheus-minute-schedule-1,28b7c6a4b54e88dd,c8a2d9b421d20e9e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:41.911 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:41.912 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:41.933 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:41.714534+08:00 2025-10-27 11:03:41.938 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,dd4aefcbf4444a60,eb5fa53962988202] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=dd4aefcbf4444a60, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:03:41.938 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,dd4aefcbf4444a60,eb5fa53962988202] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=dd4aefcbf4444a60, duration=0 ms] 2025-10-27 11:03:42.468 WARN 2458 --- [ocp-async-9,e844bcaa4262fd03,cdb792d3e502c01b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:42.468 INFO 2458 --- [ocp-async-2,85dcab792c724d40,65143387d90f3e77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:43.296 INFO 2458 --- [ocp-schedules-11,ec78cbefa6f631a4,29360ed4f7d1a11c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:43.300 INFO 2458 --- [ocp-schedules-11,ec78cbefa6f631a4,29360ed4f7d1a11c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:43.526 INFO 2458 --- [ocp-schedules-24,f6ee7d8cd2b05208,152aab2b995d53c9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:03:43.526 INFO 2458 --- [ocp-schedules-24,f6ee7d8cd2b05208,152aab2b995d53c9] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:03:43.534 INFO 2458 --- [ocp-schedules-24,f6ee7d8cd2b05208,152aab2b995d53c9] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:03:43.534 INFO 2458 --- [ocp-schedules-24,f6ee7d8cd2b05208,152aab2b995d53c9] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:03:43.941 INFO 2458 --- [ocp-schedules-15,c4345f550f6175e9,2397bd7b246179cf] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:03:44.468 INFO 2458 --- [ocp-async-5,fedeb8e613c1ccd6,75608fe5d632da4a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:44.468 WARN 2458 --- [ocp-async-8,d065a75c298d03f8,cd9e90a3f0026f6c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:44.707 INFO 2458 --- [prometheus-minute-schedule-1,3d140e4c9ebdc6e8,78a415cb2839255f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:44.716 INFO 2458 --- [prometheus-minute-schedule-1,3d140e4c9ebdc6e8,78a415cb2839255f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:44.912 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:44.913 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:45.102 INFO 2458 --- [prometheus-second-schedule-1,cc38d65e2899e5ff,28050132310ffedc] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:03:45.661 INFO 2458 --- [ocp-schedules-11,70271ac567deeaa5,2dcee30efa60546a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:46.468 INFO 2458 --- [ocp-async-7,83a067cc1d5d5846,21ff8cf189bd470b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:46.468 WARN 2458 --- [ocp-async-3,ef59952140adf700,2166e7b50b2563db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:46.559 INFO 2458 --- [ocp-schedules-15,315c98c9649aa56e,4899762f9a24b203] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:03:46.567 INFO 2458 --- [ocp-schedules-15,315c98c9649aa56e,4899762f9a24b203] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:46.567 INFO 2458 --- [ocp-schedules-15,315c98c9649aa56e,4899762f9a24b203] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:03:46.919 INFO 2458 --- [ocp-schedules-21,7cc06372996d8333,fe54e2ada1ad0d95] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:46.960 WARN 2458 --- [scanner-112123386-1,7cc06372996d8333,945e3a0068eea522] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:46.995 INFO 2458 --- [alarm-detect-7,7cc06372996d8333,b5c612b5d17e2000] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534156 ~ 1761534216 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534180, value=202514.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202514.56, values={1761534180=202514.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:03:47.061 INFO 2458 --- [alarm-detect-5,7cc06372996d8333,757b617e781cc5c9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534157 ~ 1761534217 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534180, value=92.933)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.933, values={1761534180=92.933}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:03:47.079 INFO 2458 --- [alarm-detect-2,7cc06372996d8333,92f345fb8943bffd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534195 ~ 1761534217 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534210, value=74.761)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534210, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534210=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:03:47.082 INFO 2458 --- [alarm-detect-6,7cc06372996d8333,75ff582929193f91] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534157 ~ 1761534217 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534180, value=202515.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202515.912, values={1761534180=202515.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:03:47.403 INFO 2458 --- [ocp-schedules-28,bc29aefe9dce3f96,655a125b4147d34b] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:03:47.403+08:00, clusters=[1, 2] 2025-10-27 11:03:47.419 INFO 2458 --- [ocp-schedules-28,bc29aefe9dce3f96,655a125b4147d34b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:03:47.422 INFO 2458 --- [ocp-schedules-28,bc29aefe9dce3f96,655a125b4147d34b] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:03:47.707 INFO 2458 --- [prometheus-minute-schedule-1,f86778801df8e2ee,77b80ab79f71e078] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:47.715 INFO 2458 --- [prometheus-minute-schedule-1,f86778801df8e2ee,77b80ab79f71e078] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:47.913 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:47.914 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:48.468 INFO 2458 --- [ocp-async-9,829267a47c9f1bfb,71acee9fe7704de4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:48.468 WARN 2458 --- [ocp-async-10,1727aae6715e4e5a,ee078f66d0247002] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:48.506 INFO 2458 --- [ocp-async-2,e81eb87468d63a80,5bf4852c7dc462bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:50.468 WARN 2458 --- [ocp-async-1,5ede4f56e623fd6e,ad3fec88448904b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:50.468 INFO 2458 --- [ocp-async-5,8a3faaa7fb473511,e0b6eecb321fec2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:50.707 INFO 2458 --- [prometheus-minute-schedule-1,7aca92cc6c792238,4c731ce029e2e7be] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:50.714 INFO 2458 --- [prometheus-minute-schedule-1,7aca92cc6c792238,4c731ce029e2e7be] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:50.915 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:50.916 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:51.410 INFO 2458 --- [ocp-schedules-29,5c2e7f051357c18c,721aff9494807189] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:51.410 INFO 2458 --- [ocp-schedules-25,355525ba4db0c005,f8ace13b068953de] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:51.410 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:03:51.410 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:03:51.410 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:03:51.411 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:03:51.411 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:03:51.412 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:03:51.412 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:03:51.413 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:03:51.413 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:03:51.413 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:03:51.425 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:51.425 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:51.425 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:51.426 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:51.426 INFO 2458 --- [ocp-schedules-5,f597430a2c2e331e,48f4d39c28be2f5c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:03:51.426 INFO 2458 --- [host-check-clockdiff1,f597430a2c2e331e,51bd3341c31a374c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:03:51.426 INFO 2458 --- [host-check-clockdiff2,f597430a2c2e331e,9c3c74d1bfb4423f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:03:51.427 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:51.427 INFO 2458 --- [host-check-clockdiff3,f597430a2c2e331e,2b6e34f7a3176261] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:03:51.427 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:51.427 INFO 2458 --- [host-check-clockdiff4,f597430a2c2e331e,3cbe3f1ef8dc336c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:03:51.427 INFO 2458 --- [host-check-clockdiff5,f597430a2c2e331e,dde25187cedfb113] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:03:51.430 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:51.433 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:51.442 INFO 2458 --- [ocp-schedules-30,2aed4482afd249ed,c238de1f6e7dcf78] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:03:51.439 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:51.442 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:51.443 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:51.443 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:51.444 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:51.444 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:51.444 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:51.451 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:51.452 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:51.452 INFO 2458 --- [host-check-clockdiff1,f597430a2c2e331e,51bd3341c31a374c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534231 0 0 , error= 2025-10-27 11:03:51.452 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:51.452 INFO 2458 --- [host-check-clockdiff5,f597430a2c2e331e,dde25187cedfb113] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534231 1 1 , error= 2025-10-27 11:03:51.452 INFO 2458 --- [host-check-clockdiff4,f597430a2c2e331e,3cbe3f1ef8dc336c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534231 0 0 , error= 2025-10-27 11:03:51.454 INFO 2458 --- [host-check-clockdiff3,f597430a2c2e331e,2b6e34f7a3176261] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534231 -1 -1 , error= 2025-10-27 11:03:51.454 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:51.457 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:51.538 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:03:51.538 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:03:51.539 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:03:51.539 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:03:51.539 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:03:51.540 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:03:51.540 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:03:51.540 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:03:51.539659170+08:00 2025-10-27 11:03:51.540 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:03:51.540740478+08:00 2025-10-27 11:03:51.541 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:03:51.541+08:00 2025-10-27 11:03:51.541 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:03:51.540+08:00 2025-10-27 11:03:51.542 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:03:51.542181886+08:00 2025-10-27 11:03:51.542 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:03:51.542+08:00 2025-10-27 11:03:51.542 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:03:51.541056745+08:00 2025-10-27 11:03:51.543 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:03:51.543 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:03:51.543 INFO 2458 --- [host-check-agent-status2,355525ba4db0c005,df783c6a48f49c77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:03:51.543 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:03:51.542+08:00 2025-10-27 11:03:51.543 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-agent-status4,355525ba4db0c005,e31138b69e06356a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:03:51.544 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:03:51.544261872+08:00 2025-10-27 11:03:51.545 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:03:51.544+08:00 2025-10-27 11:03:51.545 INFO 2458 --- [ocp-schedules-6,e1a1e218d2c1cdf7,4ec9215de80604b3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status1,355525ba4db0c005,35eae78fa0049e2f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:03:51.546 INFO 2458 --- [host-check-agent-status5,355525ba4db0c005,cd9ae71874b1d66d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:03:51.548 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:03:51.548 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:03:51.548 INFO 2458 --- [host-check-agent-status3,355525ba4db0c005,5e3fb06372b28c09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:03:51.549 INFO 2458 --- [ocp-schedules-28,a48b74a997403989,050442b762c22cb0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:03:51.554 INFO 2458 --- [ocp-schedules-12,9e9b300eca167c53,7c16a83bf95255d2] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:03:51.557 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:03:51.541+08:00 and status AVAILABLE 2025-10-27 11:03:51.557 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:03:51.540+08:00 and status AVAILABLE 2025-10-27 11:03:51.560 INFO 2458 --- [ocp-schedules-6,e1a1e218d2c1cdf7,4ec9215de80604b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:51.561 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:03:51.542+08:00 and status AVAILABLE 2025-10-27 11:03:51.561 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:03:51.544+08:00 and status AVAILABLE 2025-10-27 11:03:51.561 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:03:51.542+08:00 and status AVAILABLE 2025-10-27 11:03:51.562 INFO 2458 --- [ocp-schedules-12,9e9b300eca167c53,7c16a83bf95255d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:51.562 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:03:51.563 INFO 2458 --- [ocp-schedules-28,a48b74a997403989,050442b762c22cb0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:51.563 INFO 2458 --- [ocp-schedules-28,a48b74a997403989,050442b762c22cb0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:03:51.565 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:03:51.566 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:03:51.567 INFO 2458 --- [ocp-schedules-19,79b6724dd1580b25,2f40ac210e9b832f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:03:51.567 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:03:51.569 INFO 2458 --- [ocp-schedules-22,9ed6e59ccd3fb437,261cc5db78b7e49d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:03:51.569 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:03:51.569 INFO 2458 --- [ocp-schedules-11,c0785d7ab5eb62ed,2c3edbd16f6cdf21] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:03:51.579 INFO 2458 --- [ocp-schedules-22,9ed6e59ccd3fb437,261cc5db78b7e49d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:03:51.580 INFO 2458 --- [ocp-schedules-11,c0785d7ab5eb62ed,2c3edbd16f6cdf21] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:03:51.592 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:03:51.592 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:03:51.594 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:51.594 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:03:51.595 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:03:51.595 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:03:51.596 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:51.597 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:51.597 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:03:51.599 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:51.605 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.605 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:51.606 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:03:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:51.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:51.641 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:03:51.642 INFO 2458 --- [ocp-schedules-6,e1a1e218d2c1cdf7,4ec9215de80604b3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:03:51.654 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:51.664 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:51.664 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:51.664 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:51.665 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:03:51.681 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:03:51.684 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:03:51.684 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:03:51.685 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:03:51.686 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:03:51.692 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.692 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.693 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.693 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.693 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.694 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:51.710 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:51.710 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.711 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:03:51.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:03:51.714009+08:00, lastProcessTime=2025-10-27T11:03:41.714534+08:00 2025-10-27 11:03:51.725 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:03:51.726 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:03:51.729 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:03:51.729 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:03:51.734 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:03:51.759 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:51.760 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:51.760 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:51.762 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:51.762 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:03:51.770 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:51.770 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.770 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.771 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.771 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.771 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.772 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:03:51.774 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:03:51.777 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:03:51.778 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:51.778 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.779 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.782 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:03:51.783 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:03:51.784 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:51.785 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:03:51.794 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:03:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:51.812 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:03:51.812 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:03:51.815 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:03:51.818 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:03:51.823 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:03:51.832 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:51.842 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:51.845 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:51.845 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:51.849 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:51.854 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:03:51.863 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:03:51.864 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:03:51.866 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:03:51.867 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:03:51.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:51.869 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:51.871 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:03:51.875 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:51.875 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.875 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:03:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:03:51.883 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:03:51.883 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:03:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:51.894 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:03:51.896 INFO 2458 --- [host-check-host-status5,5c2e7f051357c18c,e9c14713b2ccd62f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:03:51.897 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:03:51.900 INFO 2458 --- [host-check-host-status3,5c2e7f051357c18c,5abc767e6d8859ee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:03:51.901 INFO 2458 --- [host-check-host-status4,5c2e7f051357c18c,ce467b60076bda3b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:03:51.901 INFO 2458 --- [host-check-host-status2,5c2e7f051357c18c,c977154e34e88da4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:03:51.912 INFO 2458 --- [host-check-host-status1,5c2e7f051357c18c,bcf520b05eed0c37] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:03:51.922 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:51.922 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:51.955 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:03:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:51.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:51.981 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:03:51.714009+08:00 2025-10-27 11:03:51.997 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:03:51.997 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:51.998 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:03:51.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:51.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:52.000 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.001 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:03:52.026 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:03:52.038 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:52.039 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:03:52.051 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:52.051 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:03:52.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:52.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:03:52.073 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:03:52.073 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:03:52.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:03:52.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:52.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.144 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.144 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.145 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:03:52.149 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:03:52.160 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:03:52.169 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:03:52.172 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:03:52.219 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:03:52.266 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:03:52.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.266 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.267 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:03:52.269 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.269 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.269 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.270 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:03:52.303 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:03:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.342 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:52.342 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:52.342 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:52.342 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:03:52.383 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:03:52.383 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:03:52.383 INFO 2458 --- [ocp-schedules-12,9e9b300eca167c53,7c16a83bf95255d2] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:03:52.471 WARN 2458 --- [ocp-async-10,2b6c65e9ace236fb,c50fb2b30e53342a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:52.471 INFO 2458 --- [ocp-async-10,90b63f246302b34c,d354e3680df4a3bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:03:53.219 INFO 2458 --- [ocp-schedules-11,45fb21e2e600ecbb,e0734dcfd8fee8b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20297,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5075,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7860} 2025-10-27 11:03:53.219 INFO 2458 --- [ocp-schedules-11,45fb21e2e600ecbb,e0734dcfd8fee8b4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:03:53.307 INFO 2458 --- [ocp-schedules-1,f70b3c862edb1ce8,97c37ad5729b86c5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:03:53.310 INFO 2458 --- [ocp-schedules-1,f70b3c862edb1ce8,97c37ad5729b86c5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:03:53.707 INFO 2458 --- [prometheus-minute-schedule-1,0dce3c99dd5dc904,cf145106c322d7a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:53.715 INFO 2458 --- [prometheus-minute-schedule-1,0dce3c99dd5dc904,cf145106c322d7a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:53.916 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:53.917 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:53.950 INFO 2458 --- [host-check-clockdiff2,f597430a2c2e331e,9c3c74d1bfb4423f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534233 -1 -1 , error= 2025-10-27 11:03:54.240 INFO 2458 --- [ocp-schedules-7,7e22be3536138409,161cdd541f7d0f66] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:03:54.468 WARN 2458 --- [ocp-async-9,69bb4d62ac172efc,a52cf14175f088c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:54.468 INFO 2458 --- [ocp-async-9,c343c114683bde88,b73d384e1c0aaa63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:55.259 INFO 2458 --- [ocp-schedules-14,f88273dc71fc41e2,a13d26331070dcb7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:03:55.266 INFO 2458 --- [ocp-schedules-14,f88273dc71fc41e2,a13d26331070dcb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:55.366 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:03:55.367 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.371 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:03:55.380 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.380 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.408 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.408 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.428 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.428 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.429 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:03:55.429 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.429 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.429 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.432 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:03:55.433 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.433 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.436 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.436 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.450 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.450 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.450 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.451 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.462 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.462 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.480 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.480 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.482 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.482 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.491 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.491 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:03:55.510 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.510 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.510 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.511 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.519 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.520 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.521 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.522 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.522 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.523 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.524 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.525 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.525 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.526 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.527 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.528 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.528 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:03:55.531 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:03:55.537 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.537 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.537 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.564 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.565 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.565 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.565 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:03:55.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:03:55.583 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:55.583 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.584 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:03:55.594 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.594 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:03:55.594 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.594 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:03:55.597 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:03:55.597 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:03:55.599 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.599 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.599 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.600 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:03:55.627 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.627 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.627 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.628 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:03:55.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:03:55.657 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.657 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.657 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.658 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:03:55.670 INFO 2458 --- [ocp-schedules-16,50581aa46d5cb0ae,91dac393f576a49d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:03:55.682 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.682 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.683 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.683 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:03:55.728 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.728 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.728 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.729 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.729 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.729 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.729 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.730 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.731 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.731 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.731 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.731 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.732 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.732 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.732 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.733 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.733 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.733 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.734 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.734 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.735 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.735 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.735 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.736 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.736 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.736 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.736 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.737 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.737 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.737 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.738 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.738 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.739 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.739 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.739 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.740 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.740 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.740 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.740 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.741 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:03:55.741 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.741 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.741 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.742 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:03:55.743 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.743 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.743 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.743 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:03:55.744 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:03:55.747 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:03:55.825 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:03:55.826 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:03:55.849 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:03:55.854 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:03:55.854 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:03:55.854 INFO 2458 --- [ocp-schedules-14,f88273dc71fc41e2,a13d26331070dcb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:03:56.469 INFO 2458 --- [ocp-async-8,c625659ba5f8bf9d,a56c6d72b91589e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:56.469 WARN 2458 --- [ocp-async-9,883ac0afd952802d,7fb33241fa4e4d63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:56.707 INFO 2458 --- [prometheus-minute-schedule-1,dfffd558edd461cd,59af7d8dcf4aa7f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:56.714 INFO 2458 --- [prometheus-minute-schedule-1,dfffd558edd461cd,59af7d8dcf4aa7f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:56.917 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:56.918 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:03:57.088 INFO 2458 --- [ocp-schedules-21,106c033143055a47,51cc52a62649a3a4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:03:57.121 WARN 2458 --- [scanner-112123639-1,106c033143055a47,387b125cabcb01ca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:03:57.207 INFO 2458 --- [alarm-detect-14,106c033143055a47,872d2af96cad29b1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534210 ~ 1761534227 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534225, value=45.984)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534225, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534225=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:03:57.585 INFO 2458 --- [ocp-schedules-25,5d979d654bbaac48,08a1a8ff451aae3e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:03:57.593 INFO 2458 --- [ocp-schedules-25,5d979d654bbaac48,08a1a8ff451aae3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:03:57.641 INFO 2458 --- [ocp-schedules-25,5d979d654bbaac48,08a1a8ff451aae3e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:03:58.469 WARN 2458 --- [ocp-async-8,504ba1eb2cf07c15,9f77998ba66d826b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:03:58.469 INFO 2458 --- [ocp-async-7,d73fc58f035826da,4d385c06730e28f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:03:58.506 INFO 2458 --- [ocp-async-10,d40875f6c65aa7cd,e124a29f4de450c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:03:58.897 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,695101b53f0c5285,97d7f5646b4c9337] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=695101b53f0c5285, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:03:58.897 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,695101b53f0c5285,97d7f5646b4c9337] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=695101b53f0c5285, duration=1 ms] 2025-10-27 11:03:59.097 INFO 2458 --- [ocp-schedules-7,11bccbc1c84b81c3,205ad45ff40b9ac5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:03:59.572 INFO 2458 --- [ocp-schedules-19,6111cfe84878b197,604ad649fb4520ff] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:03:59.574 INFO 2458 --- [ocp-schedules-19,6111cfe84878b197,604ad649fb4520ff] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:03:59.707 INFO 2458 --- [prometheus-minute-schedule-1,50fcd9b640a55912,fca96eeac61c546c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534240, interval=60 2025-10-27 11:03:59.714 INFO 2458 --- [prometheus-minute-schedule-1,50fcd9b640a55912,fca96eeac61c546c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:03:59.918 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:03:59.919 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:00.469 INFO 2458 --- [ocp-async-1,f56c72842d86e1e4,96cdef22f2f4f613] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:00.469 WARN 2458 --- [ocp-async-2,f1d4c5fac1cfe121,e43d3f4990a347a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:01.715576+08:00, lastProcessTime=2025-10-27T11:03:51.714009+08:00 2025-10-27 11:04:01.787 INFO 2458 --- [ocp-schedules-17,18ed700a6a1ba756,cce01e0be8d52580] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:04:01.794 INFO 2458 --- [ocp-schedules-17,18ed700a6a1ba756,cce01e0be8d52580] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:01.841 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:01.842 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:04:01.851 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:01.851 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:01.851 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:01.851 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:01.851 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:01.852 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:04:01.878 INFO 2458 --- [ocp-schedules-17,18ed700a6a1ba756,cce01e0be8d52580] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:04:01.950 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:01.715576+08:00 2025-10-27 11:04:02.469 INFO 2458 --- [ocp-async-3,6ba35528bb9a99b4,14843cc5ad423c1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:02.469 WARN 2458 --- [ocp-async-5,44e11f58481867f9,f82bacea78a29162] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:02.707 INFO 2458 --- [prometheus-minute-schedule-1,70f07da740e19754,ca03f4e2f4f57187] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:02.717 INFO 2458 --- [prometheus-minute-schedule-1,70f07da740e19754,ca03f4e2f4f57187] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:02.919 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:02.919 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:03.318 INFO 2458 --- [ocp-schedules-7,89c350e2d06c4fe7,7809bff7f1184669] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:03.322 INFO 2458 --- [ocp-schedules-7,89c350e2d06c4fe7,7809bff7f1184669] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:04.469 WARN 2458 --- [ocp-async-6,32476afe6aeaaf35,903dd81ed932d51a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:04.469 INFO 2458 --- [ocp-async-6,382d53c66a4aef51,6632883dfc9c84f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:05.677 INFO 2458 --- [ocp-schedules-1,a358b4d690d577b3,7084934ca52e2cbb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:05.707 INFO 2458 --- [prometheus-minute-schedule-1,1af494a4d5102f59,fd8d7856ae736f8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:05.714 INFO 2458 --- [prometheus-minute-schedule-1,1af494a4d5102f59,fd8d7856ae736f8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:05.919 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:05.920 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:04:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:06.003 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:04:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:04:06.075 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:04:06.120 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:04:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:04:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:04:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:04:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:04:06.145 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:04:06.146 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:04:06.146 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:04:06.146 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:04:06.146 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:04:06.146 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:04:06.147 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:06.147 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:04:06.149 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:06.149 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:04:06.219 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:01:30, start=2025-10-27T10:57:30, end=2025-10-27T11:00:00, updates=232, granule=120000ms, delay=240000ms 2025-10-27 11:04:06.231 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:01:30, target=2025-10-27T11:00:00, granule=120000ms, delay=90000ms) 2025-10-27 11:04:06.231 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:00:00, target=2025-10-27T10:52:30, granule=600000ms, delay=450000ms) 2025-10-27 11:04:06.231 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=227ms, period=120000ms 2025-10-27 11:04:06.469 WARN 2458 --- [ocp-async-10,84e9eb177a5b71d3,d860eb7ba9e5d205] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:06.469 INFO 2458 --- [ocp-async-10,74f888ee3fe6ab9c,42434a22f02017ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:06.570 INFO 2458 --- [ocp-schedules-30,f367861e6c6db8df,b678c6a18d0e79e6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:04:06.579 INFO 2458 --- [ocp-schedules-30,f367861e6c6db8df,b678c6a18d0e79e6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:06.579 INFO 2458 --- [ocp-schedules-30,f367861e6c6db8df,b678c6a18d0e79e6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:07.213 INFO 2458 --- [ocp-schedules-7,e5f2e8571ca125b7,f9d7619a2891eefe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:07.269 WARN 2458 --- [scanner-112123819-1,e5f2e8571ca125b7,d0c5963c9cfd41dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:08.469 INFO 2458 --- [ocp-async-10,ea5db858481fa804,01d1a91710bc3bc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:08.470 WARN 2458 --- [ocp-async-8,0334678cb3961c3a,4958eb2523c7fcca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:08.506 INFO 2458 --- [ocp-async-7,6dafe9d1ce073a2d,b6740eb6d561d5a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:08.707 INFO 2458 --- [prometheus-minute-schedule-1,54f3f008062077fb,e2ea758cdc9e91c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:08.715 INFO 2458 --- [prometheus-minute-schedule-1,54f3f008062077fb,e2ea758cdc9e91c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:08.920 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:08.921 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:10.018 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:04:10.021 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:04:10.096 INFO 2458 --- [Check obproxy1,38d11330ab55ed49,a5f668ca120b1319] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:04:10.096 INFO 2458 --- [Collect obproxy1,08853ce3d24b7997,0afbafabf3fc67c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:04:10.096 INFO 2458 --- [Check obproxy1,38d11330ab55ed49,a5f668ca120b1319] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:04:10.096 INFO 2458 --- [Collect obproxy1,08853ce3d24b7997,0afbafabf3fc67c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:04:10.126 INFO 2458 --- [Collect obproxy1,08853ce3d24b7997,0afbafabf3fc67c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:04:10.129 INFO 2458 --- [Check obproxy1,38d11330ab55ed49,a5f668ca120b1319] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:04:10.130 INFO 2458 --- [Collect obproxy1,08853ce3d24b7997,0afbafabf3fc67c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:04:10.132 INFO 2458 --- [Check obproxy1,38d11330ab55ed49,a5f668ca120b1319] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:04:10.132 INFO 2458 --- [Check obproxy1,38d11330ab55ed49,a5f668ca120b1319] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:04:10.133 INFO 2458 --- [Collect obproxy1,08853ce3d24b7997,0afbafabf3fc67c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:04:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:04:10.470 WARN 2458 --- [ocp-async-4,8f03a423bdca20f7,a5550edf74a7252e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:10.470 INFO 2458 --- [ocp-async-2,1950f6be225f8c13,9f1981f0dc4c76b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:11.707 INFO 2458 --- [prometheus-minute-schedule-1,ba02174b4e086d38,c7a883d11fd8d496] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:11.714 INFO 2458 --- [prometheus-minute-schedule-1,ba02174b4e086d38,c7a883d11fd8d496] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:11.714503+08:00, lastProcessTime=2025-10-27T11:04:01.715576+08:00 2025-10-27 11:04:11.921 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:11.922 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:11.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:11.714503+08:00 2025-10-27 11:04:12.470 WARN 2458 --- [ocp-async-2,49f893e09c36ecd7,b0f612bb833baa44] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:12.470 INFO 2458 --- [ocp-async-2,49dc668707109044,90a64fde3ac43aaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:12.668 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657} 2025-10-27 11:04:12.676 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClust 2025-10-27 11:04:12.677 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:12.699 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:12.699 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:04:12.699 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:12.836 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447852836000, endTimeUs=1761534132657000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:04:12.836Z, endTime=2025-10-27T03:02:12.657Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:04:12.851 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:04:12.851 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:04:12.875 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:04:12.875 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534132657000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:04:12.957 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:04:12.957 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:04:12.984 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:04:12.984 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534132657000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:04:12.992 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:04:12.992 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:04:12.836, end=2025-10-27T11:02:12.657}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:04:12.992 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:12.994 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:12.994 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:04:12.994 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:12.997 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:12.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:12.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:04:12.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:13.001 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:13.004 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:13.004 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:04:13.004 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:13.011 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:13.012 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:13.012 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:04:13.012 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:13.015 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:04:13.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:11.995, end=2025-10-27T11:02:12.657},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:04:13.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:04:13.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:04:13.018 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:04:13.329 INFO 2458 --- [ocp-schedules-6,e39a6c3c5acfe376,8856470479b2c3f8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:13.333 INFO 2458 --- [ocp-schedules-6,e39a6c3c5acfe376,8856470479b2c3f8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:13.422 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=1000, affectRows=1000 2025-10-27 11:04:13.427 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=994, validCount=992 2025-10-27 11:04:13.501 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=1000, affectRows=1000 2025-10-27 11:04:13.505 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=999, validCount=995 2025-10-27 11:04:13.517 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534180, nodeCount=817, affectRows=817 2025-10-27 11:04:13.520 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587936) Duplicate entry '835370603008958464-1761534120' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587936) Duplicate entry '835370603008958464-1761534120' for key 'PRIMARY' 2025-10-27 11:04:13.556 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=1000, affectRows=1000 2025-10-27 11:04:13.560 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=992, validCount=989 2025-10-27 11:04:13.623 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=1000, affectRows=1000 2025-10-27 11:04:13.627 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=1000, validCount=995 2025-10-27 11:04:13.644 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=75, affectRows=75 2025-10-27 11:04:13.645 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=74, validCount=74 2025-10-27 11:04:13.681 INFO 2458 --- [ocp-schedules-26,2283b4f22ab9bb98,d3f943f346f69e81] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:04:13.681 INFO 2458 --- [ocp-schedules-26,2283b4f22ab9bb98,d3f943f346f69e81] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:04:13.689 INFO 2458 --- [ocp-schedules-26,2283b4f22ab9bb98,d3f943f346f69e81] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:04:13.689 INFO 2458 --- [ocp-schedules-26,2283b4f22ab9bb98,d3f943f346f69e81] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:04:13.809 INFO 2458 --- [ocp-schedules-22,821cac44b8a95f69,04ded01de7d7aac9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:04:13.809 INFO 2458 --- [ocp-schedules-22,821cac44b8a95f69,04ded01de7d7aac9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:04:13.877 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:04:13.878 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:04:13.878 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:04:13.878 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:04:13.879 INFO 2458 --- [Collect all tenants major compactions5,821cac44b8a95f69,2d9c1867efdb134d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:04:13.879 INFO 2458 --- [Collect all tenants major compactions6,821cac44b8a95f69,03fd64964e19a5fa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:04:13.939 INFO 2458 --- [Collect all tenants major compactions5,821cac44b8a95f69,2d9c1867efdb134d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.940 INFO 2458 --- [Collect all tenants major compactions6,821cac44b8a95f69,03fd64964e19a5fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.940 INFO 2458 --- [Collect all tenants major compactions5,821cac44b8a95f69,2d9c1867efdb134d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:04:13.941 INFO 2458 --- [Collect all tenants major compactions6,821cac44b8a95f69,03fd64964e19a5fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:04:13.944 INFO 2458 --- [Collect all tenants major compactions6,821cac44b8a95f69,03fd64964e19a5fa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:04:13.945 INFO 2458 --- [Collect all tenants major compactions5,821cac44b8a95f69,2d9c1867efdb134d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:04:13.958 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.958 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.958 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.958 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.959 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.960 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:13.961 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:13.961 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:13.961 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:04:13.961 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:04:13.962 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:04:13.964 INFO 2458 --- [Collect all tenants major compactions2,821cac44b8a95f69,11e5876d41c50768] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:04:13.964 INFO 2458 --- [Collect all tenants major compactions1,821cac44b8a95f69,7ad1afbf28905738] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:04:13.965 INFO 2458 --- [Collect all tenants major compactions4,821cac44b8a95f69,1c9a232da45933aa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:04:13.965 INFO 2458 --- [Collect all tenants major compactions3,821cac44b8a95f69,17459160639cb819] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:04:13.966 INFO 2458 --- [ocp-schedules-22,821cac44b8a95f69,04ded01de7d7aac9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:04:14.094 INFO 2458 --- [ocp-schedules-23,85af4db0ba8d4ab9,def1cf1b9686bd4c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:04:14.094+08:00, clusters=[1, 2] 2025-10-27 11:04:14.097 INFO 2458 --- [ocp-schedules-23,85af4db0ba8d4ab9,def1cf1b9686bd4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:04:14.097 INFO 2458 --- [ocp-schedules-23,85af4db0ba8d4ab9,def1cf1b9686bd4c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:04:14.470 WARN 2458 --- [ocp-async-8,6449359ca9f209f5,13a928dd1f73ca46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:14.470 INFO 2458 --- [ocp-async-1,a1d4180cadee58ea,5ff66e8888f13d83] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:14.707 INFO 2458 --- [prometheus-minute-schedule-1,9b6c7dcf2794b234,128f37cf3909a6af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:14.715 INFO 2458 --- [prometheus-minute-schedule-1,9b6c7dcf2794b234,128f37cf3909a6af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:14.922 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:14.923 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:15.685 INFO 2458 --- [ocp-schedules-8,6c26c32175311977,f7ce779601d109fe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:16.470 WARN 2458 --- [ocp-async-3,33d72a2d8a2bb224,e1db6919d91d91fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:16.470 INFO 2458 --- [ocp-async-7,91da93fcb35731a9,362586e5bd3d992d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:17.315 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534180, nodeCount=1000, affectRows=1000 2025-10-27 11:04:17.319 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534180, nodeCount=1000, validCount=987 2025-10-27 11:04:17.401 INFO 2458 --- [ocp-schedules-11,ef238259d480c773,c4e8d748e801fe17] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:17.412 WARN 2458 --- [scanner-112125590-1,ef238259d480c773,c8132b943238ae37] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:17.515 INFO 2458 --- [alarm-detect-11,ef238259d480c773,97fa2693d16f4eeb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534225 ~ 1761534247 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534240, value=50.723)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534240, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534240=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:04:17.707 INFO 2458 --- [prometheus-minute-schedule-1,2aae9417953e3c28,77814dc7a72452e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:17.715 INFO 2458 --- [prometheus-minute-schedule-1,2aae9417953e3c28,77814dc7a72452e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:17.923 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:17.924 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:18.471 WARN 2458 --- [ocp-async-9,702e963cd75ed52b,2dab8f56f720adb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:18.471 INFO 2458 --- [ocp-async-6,0ac1eefbcf95ac05,e1ea1e2e27afbe86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:18.506 INFO 2458 --- [ocp-async-2,6dfc8cbf528fc874,7845b0c31bbc53ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:18.970 INFO 2458 --- [ocp-schedules-25,e0e125789314b6f4,438a6d14edae4e7c] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:04:20.470 INFO 2458 --- [ocp-async-8,1e422c4a46a8ad83,14f7633bb619b0fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:20.470 WARN 2458 --- [ocp-async-1,2a0c65874d9f0f0f,7e2324f0ba0f9d32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:20.707 INFO 2458 --- [prometheus-minute-schedule-1,4469032cc7c8900f,c87ea53e5a7702d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:20.715 INFO 2458 --- [prometheus-minute-schedule-1,4469032cc7c8900f,c87ea53e5a7702d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:20.924 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:20.925 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:21.409 INFO 2458 --- [ocp-schedules-8,beae6b541d92d0f3,efe0064f4f9da0c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:21.409 INFO 2458 --- [ocp-schedules-7,eaa0aa7307b53291,1933634ca5bbf75a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:21.409 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:04:21.409 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:04:21.410 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:04:21.411 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:04:21.413 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:04:21.425 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:21.425 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:21.425 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:21.426 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:21.426 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:21.427 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:21.427 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:21.427 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:21.430 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:21.430 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:21.431 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:21.431 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:21.432 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:21.432 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:21.432 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:21.432 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:21.432 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:21.433 INFO 2458 --- [ocp-schedules-27,5b85e70f193ecf76,008f2451b4f79313] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:21.433 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:21.434 INFO 2458 --- [host-check-clockdiff1,5b85e70f193ecf76,ff02a490e66c9ba9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:04:21.435 INFO 2458 --- [host-check-clockdiff2,5b85e70f193ecf76,c01cb7d65faa5e2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:04:21.435 INFO 2458 --- [host-check-clockdiff3,5b85e70f193ecf76,607a74c56be4138a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:04:21.436 INFO 2458 --- [host-check-clockdiff4,5b85e70f193ecf76,1744813209ce9f35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:04:21.437 INFO 2458 --- [host-check-clockdiff5,5b85e70f193ecf76,e7da4f7117e418e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:04:21.438 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:21.458 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:21.459 INFO 2458 --- [host-check-clockdiff3,5b85e70f193ecf76,607a74c56be4138a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534261 -1 -1 , error= 2025-10-27 11:04:21.460 INFO 2458 --- [ocp-schedules-24,15fa22dc4f920460,9104602665df8ee3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:04:21.468 INFO 2458 --- [host-check-clockdiff1,5b85e70f193ecf76,ff02a490e66c9ba9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534261 0 0 , error= 2025-10-27 11:04:21.482 INFO 2458 --- [host-check-clockdiff2,5b85e70f193ecf76,c01cb7d65faa5e2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534261 -1 -1 , error= 2025-10-27 11:04:21.545 INFO 2458 --- [ocp-schedules-5,60bc0fac897a4a30,9ee3036572013952] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:04:21.549 INFO 2458 --- [ocp-schedules-22,fb8a90a179b1edd2,62b647c08b31772e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:04:21.554 INFO 2458 --- [ocp-schedules-25,807146edd1a3245e,b93a7144ebadd1f6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:04:21.556 INFO 2458 --- [ocp-schedules-22,fb8a90a179b1edd2,62b647c08b31772e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:21.557 INFO 2458 --- [ocp-schedules-22,fb8a90a179b1edd2,62b647c08b31772e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:04:21.561 INFO 2458 --- [ocp-schedules-25,807146edd1a3245e,b93a7144ebadd1f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:21.566 INFO 2458 --- [ocp-schedules-13,8c40874cdeca11f5,51ffb4d175ff97dd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:04:21.576 INFO 2458 --- [ocp-schedules-5,60bc0fac897a4a30,9ee3036572013952] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:21.637 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:04:21.639 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:04:21.637740762+08:00 2025-10-27 11:04:21.639 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:04:21.639+08:00 2025-10-27 11:04:21.646 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:04:21.647 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:04:21.648 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.649 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:21.649 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:04:21.648361576+08:00 2025-10-27 11:04:21.650 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:04:21.649+08:00 2025-10-27 11:04:21.650 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:04:21.652 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:04:21.652 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:04:21.652 INFO 2458 --- [host-check-agent-status2,eaa0aa7307b53291,095b4b561966c241] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:04:21.654 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:04:21.639+08:00 and status AVAILABLE 2025-10-27 11:04:21.657 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:04:21.659 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:04:21.659486388+08:00 2025-10-27 11:04:21.660 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:04:21.659+08:00 2025-10-27 11:04:21.661 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:04:21.665 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:04:21.649+08:00 and status AVAILABLE 2025-10-27 11:04:21.666 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:04:21.668 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:04:21.668 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:04:21.668 INFO 2458 --- [host-check-agent-status5,eaa0aa7307b53291,1d280f7dee00ef98] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:04:21.669 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:04:21.659+08:00 and status AVAILABLE 2025-10-27 11:04:21.671 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:04:21.674 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:04:21.688 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:04:21.689 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:04:21.689608970+08:00 2025-10-27 11:04:21.690 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:04:21.689+08:00 2025-10-27 11:04:21.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:21.692 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:04:21.694 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.694 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.694 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.694 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.695 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.695 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:21.697 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:04:21.699 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:04:21.700 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:04:21.700 INFO 2458 --- [host-check-agent-status1,eaa0aa7307b53291,84fbbf5c498d07e7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:04:21.700 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:04:21.700 INFO 2458 --- [ocp-schedules-5,60bc0fac897a4a30,9ee3036572013952] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:04:21.701 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:04:21.689+08:00 and status AVAILABLE 2025-10-27 11:04:21.703 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:04:21.703 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:04:21.704 INFO 2458 --- [host-check-agent-status3,eaa0aa7307b53291,d2cee89ccfabe2de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:04:21.704 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:21.705 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:04:21.705 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:04:21.706 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:04:21.707 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:21.707 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:04:21.706737647+08:00 2025-10-27 11:04:21.708 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:04:21.707+08:00 2025-10-27 11:04:21.708 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:21.708 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:04:21.708 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:04:21.708 INFO 2458 --- [host-check-agent-status4,eaa0aa7307b53291,f5c2c5754d6a6ec5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:04:21.711 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:21.714 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:21.718 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:04:21.707+08:00 and status AVAILABLE 2025-10-27 11:04:21.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:21.718858+08:00, lastProcessTime=2025-10-27T11:04:11.714503+08:00 2025-10-27 11:04:21.722 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:21.725 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:04:21.729 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:21.732 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:21.737 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.737 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.738 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.738 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.738 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.738 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:21.751 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:21.754 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:21.772 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:21.773 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.774 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:04:21.780 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:21.789 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:04:21.795 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:04:21.799 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:21.803 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:04:21.820 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:21.836 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:04:21.837 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:04:21.842 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:04:21.845 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:04:21.846 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:04:21.854 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:21.854 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.854 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:04:21.871 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:04:21.872 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:04:21.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.875 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:21.876 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:04:21.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:21.884 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:21.884 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:21.886 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:21.901 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:04:21.908 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:04:21.908 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:04:21.912 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:21.926 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:21.941 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:04:21.941 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:04:21.942 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:04:21.944 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:04:21.945 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:04:21.958 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:21.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.972 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.973 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:21.974 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:04:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.975 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:21.977 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:21.978 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:04:21.979 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:21.985 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:21.986 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:04:21.993 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:21.995 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:04:21.996 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:04:21.998 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:04:22.007 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:04:22.008 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:04:22.008 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:22.014 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:21.718858+08:00 2025-10-27 11:04:22.018 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:22.018 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:22.021 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:22.023 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:04:22.027 INFO 2458 --- [host-check-host-status2,beae6b541d92d0f3,e4ee5f36ce64485d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:04:22.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:22.036 INFO 2458 --- [host-check-host-status5,beae6b541d92d0f3,3374c45e6544b14a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:04:22.042 INFO 2458 --- [host-check-host-status3,beae6b541d92d0f3,42f49f5a63f247ca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:04:22.047 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:04:22.054 INFO 2458 --- [host-check-host-status4,beae6b541d92d0f3,fb7b39a680fa465f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:04:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:22.077 INFO 2458 --- [host-check-host-status1,beae6b541d92d0f3,52e969d8e1fb7a1a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:04:22.077 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:04:22.116 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.116 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:22.120 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:04:22.120 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.121 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:04:22.123 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.124 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:04:22.141 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:04:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:22.163 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:22.164 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.165 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:04:22.191 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.191 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:22.191 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:04:22.191 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:04:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:22.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:22.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.267 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.267 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:22.271 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:04:22.280 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:04:22.290 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:22.293 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:22.340 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:04:22.381 INFO 2458 --- [host-check-clockdiff5,5b85e70f193ecf76,e7da4f7117e418e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534262 0 0 , error= 2025-10-27 11:04:22.391 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:04:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:04:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.394 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:04:22.427 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:04:22.464 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:22.464 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.464 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.465 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:22.465 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:22.465 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:22.466 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:04:22.470 INFO 2458 --- [ocp-async-3,2df1e39296a7f546,55248974252b6944] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:22.470 WARN 2458 --- [ocp-async-4,d02e636be6e72313,cac1414ead0503e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:22.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:04:22.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:04:22.509 INFO 2458 --- [ocp-schedules-25,807146edd1a3245e,b93a7144ebadd1f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:04:23.228 INFO 2458 --- [host-check-clockdiff4,5b85e70f193ecf76,1744813209ce9f35] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534263 0 0 , error= 2025-10-27 11:04:23.340 INFO 2458 --- [ocp-schedules-9,90ffaa0da6079cd0,b5997470d03eb25b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:23.344 INFO 2458 --- [ocp-schedules-9,90ffaa0da6079cd0,b5997470d03eb25b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:23.707 INFO 2458 --- [prometheus-minute-schedule-1,ced5854ec8dd8ec6,8a56b4f6e8b0568d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:23.715 INFO 2458 --- [prometheus-minute-schedule-1,ced5854ec8dd8ec6,8a56b4f6e8b0568d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:23.925 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:23.925 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:24.471 INFO 2458 --- [ocp-async-9,e4c8e47ec49a1db7,39af9c7c94ddfecd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:24.471 WARN 2458 --- [ocp-async-5,2ce78aeb950ca443,16ce6a0d66913392] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:25.692 INFO 2458 --- [ocp-schedules-4,0ae2089ef384472e,e2eec87a943803a7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:26.471 INFO 2458 --- [ocp-async-10,56fb8b569b396eb8,dd979455ddb3c0a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:26.471 WARN 2458 --- [ocp-async-1,67b338c35245580d,9a77ef5019283315] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:26.582 INFO 2458 --- [ocp-schedules-24,380b6de998f393b4,3d371ca9655a8d62] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:04:26.590 INFO 2458 --- [ocp-schedules-24,380b6de998f393b4,3d371ca9655a8d62] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:26.590 INFO 2458 --- [ocp-schedules-24,380b6de998f393b4,3d371ca9655a8d62] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:26.707 INFO 2458 --- [prometheus-minute-schedule-1,343cffa86fab96d8,13e8af4fea100055] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:26.714 INFO 2458 --- [prometheus-minute-schedule-1,343cffa86fab96d8,13e8af4fea100055] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:26.926 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:26.926 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:27.521 INFO 2458 --- [ocp-schedules-19,b17aed5a459d9e24,51ce0fb18c164bdd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:27.566 WARN 2458 --- [scanner-112125803-1,b17aed5a459d9e24,cddb802e911606c5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:27.696 INFO 2458 --- [alarm-detect-12,b17aed5a459d9e24,4be8a23c914240e3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534240 ~ 1761534257 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534255, value=67.245)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534255, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534255=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:04:28.471 WARN 2458 --- [ocp-async-8,05926b561b3e23da,e21c6ebdc9464437] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:28.471 INFO 2458 --- [ocp-async-7,9cc62482fe15c94e,a92814a1beab3a6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:28.506 INFO 2458 --- [ocp-async-3,255ecea0cc0c7334,ad19ef0dc95bda93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:28.814 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:04:28.814 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:04:28.816 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:04:28.816 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:04:28.817 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:04:28.817 INFO 2458 --- [ocp-schedules-17,1a17bd150643e230,59ceb054f89d99d2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:04:29.287 INFO 2458 --- [ocp-schedules-22,897e2ab87c90e14f,02774c3d052188ed] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:04:29.295 INFO 2458 --- [ocp-schedules-22,897e2ab87c90e14f,02774c3d052188ed] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:04:29.295 INFO 2458 --- [ocp-schedules-22,897e2ab87c90e14f,02774c3d052188ed] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:04:29.707 INFO 2458 --- [prometheus-minute-schedule-1,d52ad523d5414059,a7acb6f5cc877677] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:29.717 INFO 2458 --- [prometheus-minute-schedule-1,d52ad523d5414059,a7acb6f5cc877677] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:29.926 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:29.927 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:30.471 WARN 2458 --- [ocp-async-6,7a8470b756ce6756,547df141dd4d94e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:30.471 INFO 2458 --- [ocp-async-6,c2477705ad7bd53a,e04c765ef2aee33f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:31.715975+08:00, lastProcessTime=2025-10-27T11:04:21.718858+08:00 2025-10-27 11:04:31.923 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:31.715975+08:00 2025-10-27 11:04:32.471 INFO 2458 --- [ocp-async-1,537c0e4124220922,cb0e0c9c3802c58a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:32.471 WARN 2458 --- [ocp-async-2,79beadea41897b88,8a0a42b77089aa34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:32.707 INFO 2458 --- [prometheus-minute-schedule-1,86b3135a08a712bc,6ab5fcb34b0505b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:32.716 INFO 2458 --- [prometheus-minute-schedule-1,86b3135a08a712bc,6ab5fcb34b0505b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:32.927 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:32.928 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:33.352 INFO 2458 --- [ocp-schedules-21,fb6b79e6d1cb704e,72b0e898c7a0ba7d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:33.356 INFO 2458 --- [ocp-schedules-21,fb6b79e6d1cb704e,72b0e898c7a0ba7d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:33.708 INFO 2458 --- [ocp-schedules-14,41a6327d9e85158c,a22a3d014a3f7237] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:04:33.708 INFO 2458 --- [ocp-schedules-14,41a6327d9e85158c,a22a3d014a3f7237] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:04:34.304 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:04:34.304 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:04:34.379 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:34.379 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:34.420 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:34.420 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:34.420 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:34.421 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:04:34.433 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:34.433 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:34.433 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:34.434 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:04:34.471 INFO 2458 --- [ocp-async-8,8e10614bc766894c,549406748cc464ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:34.471 WARN 2458 --- [ocp-async-4,34e3e55fe7456bc6,7b74686103c2de5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:34.495 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:34.496 INFO 2458 --- [ocp-schedules-1,b6dcbc8b6cc18431,9c2bcbe172982a9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:04:35.699 INFO 2458 --- [ocp-schedules-9,1223d0efb33df793,086afcd93345e1cd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:35.707 INFO 2458 --- [prometheus-minute-schedule-1,7e879ea90f1df429,5892684586c3db93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:35.713 INFO 2458 --- [prometheus-minute-schedule-1,7e879ea90f1df429,5892684586c3db93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:35.928 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:35.929 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:36.471 WARN 2458 --- [ocp-async-9,911643171be39a7b,1863bd4f3609c9af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:36.471 INFO 2458 --- [ocp-async-3,fa975b8bcc377179,472ccd27ce5395dd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:36.642 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:36.642 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:04:36.642 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:04:36.642 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:04:36.642 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:04:36.642 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:04:36.655 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:36.655 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:36.655 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:36.655 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:36.656 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:36.659 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:36.659 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:36.659 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:36.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:36.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:36.720 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:04:36.722 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:04:36.724 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:04:36.726 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:04:36.726 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:04:36.730 INFO 2458 --- [ocp-schedules-3,361232fe948d487f,1e8f95a00669e50a] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:04:37.702 INFO 2458 --- [ocp-schedules-11,22df0bd09c0a0477,21a9a25dbf6eeac2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:37.708 INFO 2458 --- [prometheus-cache-monitor1,bc98578c129a5fe5,0b92f3818252a272] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20298,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5074,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7860} 2025-10-27 11:04:37.708 INFO 2458 --- [prometheus-cache-monitor1,bc98578c129a5fe5,0b92f3818252a272] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:04:37.733 WARN 2458 --- [scanner-112126019-1,22df0bd09c0a0477,6b4c58865bd6a0dc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:38.472 WARN 2458 --- [ocp-async-6,72e01c2052ad2ac8,fc824bcdac205309] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:38.472 INFO 2458 --- [ocp-async-2,b130673b6cbe1f94,76c5dc916b591966] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:38.507 INFO 2458 --- [ocp-async-1,903b6591db6528a4,5e9419530c5144c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:38.565 INFO 2458 --- [ocp-schedules-18,a031e9d831434efa,ea8b854b2e468287] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:04:38.571 INFO 2458 --- [ocp-schedules-18,a031e9d831434efa,ea8b854b2e468287] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:38.571 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:04:38.571 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:04:38.571 INFO 2458 --- [ocp-schedules-18,a031e9d831434efa,ea8b854b2e468287] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:04:38.707 INFO 2458 --- [prometheus-minute-schedule-1,d24cdcf4571b0fd6,a1d69e48ff0ec083] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:38.717 INFO 2458 --- [prometheus-minute-schedule-1,d24cdcf4571b0fd6,a1d69e48ff0ec083] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:38.929 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:38.930 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:40.014 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:04:40.018 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:04:40.054 INFO 2458 --- [Check obproxy1,5921675bc3990940,00ec50c73482db45] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:04:40.054 INFO 2458 --- [Collect obproxy1,47095cb2277ae0ab,3909b634acbb04d1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:04:40.055 INFO 2458 --- [Check obproxy1,5921675bc3990940,00ec50c73482db45] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:04:40.055 INFO 2458 --- [Collect obproxy1,47095cb2277ae0ab,3909b634acbb04d1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:04:40.084 INFO 2458 --- [Check obproxy1,5921675bc3990940,00ec50c73482db45] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:04:40.085 INFO 2458 --- [Collect obproxy1,47095cb2277ae0ab,3909b634acbb04d1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:04:40.088 INFO 2458 --- [Check obproxy1,5921675bc3990940,00ec50c73482db45] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:04:40.088 INFO 2458 --- [Check obproxy1,5921675bc3990940,00ec50c73482db45] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:04:40.089 INFO 2458 --- [Collect obproxy1,47095cb2277ae0ab,3909b634acbb04d1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:04:40.092 INFO 2458 --- [Collect obproxy1,47095cb2277ae0ab,3909b634acbb04d1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:04:40.167 INFO 2458 --- [ocp-schedules-16,8706e017033ed70b,ea227cab533aa48d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:04:40.167 INFO 2458 --- [ocp-schedules-16,8706e017033ed70b,ea227cab533aa48d] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:04:40.298 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5e7307163790c29b,8456fbb15d5ee078] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=5e7307163790c29b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:04:40.299 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5e7307163790c29b,8456fbb15d5ee078] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=5e7307163790c29b, duration=1 ms] 2025-10-27 11:04:40.472 WARN 2458 --- [ocp-async-4,c15191b2ad7613b1,b1bfafce1c683ef4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:40.472 INFO 2458 --- [ocp-async-4,b1626990f9c6718d,ff1c8edebac2c6e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:41.707 INFO 2458 --- [prometheus-minute-schedule-1,f3413e6154bba57f,9982563bdf8ddd24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:41.714 INFO 2458 --- [prometheus-minute-schedule-1,f3413e6154bba57f,9982563bdf8ddd24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:41.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:41.720051+08:00, lastProcessTime=2025-10-27T11:04:31.715975+08:00 2025-10-27 11:04:41.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:41.720051+08:00 2025-10-27 11:04:41.930 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:41.930 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:42.472 WARN 2458 --- [ocp-async-5,3488bb1f88b4a762,a934ca608df87e1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:42.472 INFO 2458 --- [ocp-async-5,dbdae2ab31bbdcba,062a36ce30d346f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:43.364 INFO 2458 --- [ocp-schedules-20,853ea59852d68870,45932256ecc78eee] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:43.368 INFO 2458 --- [ocp-schedules-20,853ea59852d68870,45932256ecc78eee] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:43.835 INFO 2458 --- [ocp-schedules-22,de9dc42a9cda40c2,7318a154b33df2db] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:04:43.835 INFO 2458 --- [ocp-schedules-22,de9dc42a9cda40c2,7318a154b33df2db] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:04:43.843 INFO 2458 --- [ocp-schedules-22,de9dc42a9cda40c2,7318a154b33df2db] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:04:43.843 INFO 2458 --- [ocp-schedules-22,de9dc42a9cda40c2,7318a154b33df2db] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:04:43.953 INFO 2458 --- [ocp-schedules-11,8e1d91d0bde9ea76,8d69303387a15556] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:04:44.472 WARN 2458 --- [ocp-async-10,326bfecd4c0f0155,55f9cbcfd3347e8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:44.472 INFO 2458 --- [ocp-async-6,edfb3bf2f924247d,bd0d501cac9c5b14] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:44.707 INFO 2458 --- [prometheus-minute-schedule-1,bc3baee216458406,f40c5d0bd52913f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:44.715 INFO 2458 --- [prometheus-minute-schedule-1,bc3baee216458406,f40c5d0bd52913f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:44.930 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:44.931 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:45.102 INFO 2458 --- [prometheus-second-schedule-1,bf194c522f083d83,f29ee565718caf9d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:04:45.439 INFO 2458 --- [ocp-schedules-4,b9a4d2af6e69c755,f862087a3c2d110e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:04:45.439 INFO 2458 --- [ocp-schedules-4,b9a4d2af6e69c755,f862087a3c2d110e] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:04:45.500 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:04:45.501 INFO 2458 --- [pool-876-thread-2,b9a4d2af6e69c755,d770e12e9387b52c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:04:45.553 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:45.553 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:45.568 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:45.568 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:45.568 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:45.569 INFO 2458 --- [pool-876-thread-2,b9a4d2af6e69c755,d770e12e9387b52c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:45.571 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761533985000000, 1761534285000000] 2025-10-27 11:04:45.572 INFO 2458 --- [pool-876-thread-2,b9a4d2af6e69c755,d770e12e9387b52c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761533985000000, 1761534285000000] 2025-10-27 11:04:45.616 WARN 2458 --- [pool-876-thread-2,b9a4d2af6e69c755,d770e12e9387b52c] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:04:45.629 INFO 2458 --- [pool-876-thread-2,b9a4d2af6e69c755,d770e12e9387b52c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:04:45.686 WARN 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:04:45.686 WARN 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:04:45.686 WARN 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:04:45.693 INFO 2458 --- [pool-876-thread-1,b9a4d2af6e69c755,897277cda2477778] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:04:45.694 INFO 2458 --- [ocp-schedules-4,b9a4d2af6e69c755,f862087a3c2d110e] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:04:45.706 INFO 2458 --- [ocp-schedules-21,ab1d01f5f44d7dee,39001a4397a24364] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:46.472 WARN 2458 --- [ocp-async-6,9dfc7dd452f9f3c0,6aa5a75c9fd82d5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:46.472 INFO 2458 --- [ocp-async-8,138ee1aef8da7856,4d11209286d66ddd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:46.593 INFO 2458 --- [ocp-schedules-17,85f1f706e7fa29ce,5e5a2a1743208f45] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:04:46.601 INFO 2458 --- [ocp-schedules-17,85f1f706e7fa29ce,5e5a2a1743208f45] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:46.601 INFO 2458 --- [ocp-schedules-17,85f1f706e7fa29ce,5e5a2a1743208f45] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:04:47.473 INFO 2458 --- [ocp-schedules-14,f706e2734c486ccf,e43431696947d367] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:04:47.473+08:00, clusters=[1, 2] 2025-10-27 11:04:47.487 INFO 2458 --- [ocp-schedules-14,f706e2734c486ccf,e43431696947d367] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:04:47.489 INFO 2458 --- [ocp-schedules-14,f706e2734c486ccf,e43431696947d367] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:04:47.707 INFO 2458 --- [prometheus-minute-schedule-1,fe519f6d071308a6,6aaab69e5a88c33b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:47.715 INFO 2458 --- [prometheus-minute-schedule-1,fe519f6d071308a6,6aaab69e5a88c33b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:47.841 INFO 2458 --- [ocp-schedules-24,1de2839d4d82f4ed,11d5150e73dd42ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:47.906 WARN 2458 --- [scanner-112126263-1,1de2839d4d82f4ed,fe816fe638379bbc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:47.931 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:47.932 INFO 2458 --- [alarm-detect-20,1de2839d4d82f4ed,38c0d068448ed815] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534217 ~ 1761534277 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534240, value=202574.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202574.56, values={1761534240=202574.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:04:47.932 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:47.983 INFO 2458 --- [alarm-detect-17,1de2839d4d82f4ed,5cf7c7f44971d9fd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534217 ~ 1761534277 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534240, value=92.942)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.942, values={1761534240=92.942}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:04:48.008 INFO 2458 --- [alarm-detect-16,1de2839d4d82f4ed,37ea59fede78214d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534217 ~ 1761534277 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534240, value=202575.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202575.912, values={1761534240=202575.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:04:48.027 INFO 2458 --- [alarm-detect-7,1de2839d4d82f4ed,95bdda39b49dddb5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534255 ~ 1761534278 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534270, value=60.395)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534270, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534270=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:04:48.472 WARN 2458 --- [ocp-async-8,500156061b9632bf,34207e18fea9616d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:48.472 INFO 2458 --- [ocp-async-4,822ef2a206e2f5b8,7e916da0a214c61e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:48.507 INFO 2458 --- [ocp-async-10,de12665c968b5bef,357cdb5d918462c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:50.472 WARN 2458 --- [ocp-async-10,06c0f425e202772b,9e27c830808a329d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:50.472 INFO 2458 --- [ocp-async-5,73d7f44b0b9edd1b,73ec233766eaf368] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:50.707 INFO 2458 --- [prometheus-minute-schedule-1,cce6a7c8517a1733,e8f1f5434b374895] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:50.717 INFO 2458 --- [prometheus-minute-schedule-1,cce6a7c8517a1733,e8f1f5434b374895] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:50.932 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:50.933 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:51.410 INFO 2458 --- [ocp-schedules-14,0aa7be348725f200,6fc3eaee0ac7769a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:51.410 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:04:51.410 INFO 2458 --- [ocp-schedules-13,37714e3fa3dcf2d3,a754d8811ffe5a97] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:51.411 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:04:51.412 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:04:51.413 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:04:51.413 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:04:51.413 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:04:51.414 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:04:51.414 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:04:51.415 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:04:51.415 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:04:51.428 INFO 2458 --- [ocp-schedules-29,473decb77577a4a3,99aa76e9770a2cc6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:04:51.428 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:51.428 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:51.429 INFO 2458 --- [host-check-clockdiff1,473decb77577a4a3,08554da5c7899355] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:04:51.429 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:51.429 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:51.429 INFO 2458 --- [host-check-clockdiff3,473decb77577a4a3,02676a3a8d7477db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:04:51.430 INFO 2458 --- [host-check-clockdiff4,473decb77577a4a3,24e0c6b24347230f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:04:51.430 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:51.430 INFO 2458 --- [host-check-clockdiff2,473decb77577a4a3,de756ff343001288] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:04:51.430 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:51.431 INFO 2458 --- [host-check-clockdiff5,473decb77577a4a3,b62ea752aea58cdc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:04:51.439 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:51.440 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:51.440 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:51.440 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:51.440 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:51.441 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:51.441 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:51.442 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:51.442 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:51.442 INFO 2458 --- [ocp-schedules-28,621b558b04ebcd43,0ab796a4af02638e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:04:51.449 INFO 2458 --- [host-check-clockdiff2,473decb77577a4a3,de756ff343001288] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534291 -1 -1 , error= 2025-10-27 11:04:51.450 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:51.451 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:51.452 INFO 2458 --- [host-check-clockdiff4,473decb77577a4a3,24e0c6b24347230f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534291 0 0 , error= 2025-10-27 11:04:51.452 INFO 2458 --- [host-check-clockdiff5,473decb77577a4a3,b62ea752aea58cdc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534291 0 0 , error= 2025-10-27 11:04:51.452 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:51.452 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:51.455 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:51.457 INFO 2458 --- [host-check-clockdiff1,473decb77577a4a3,08554da5c7899355] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534291 0 0 , error= 2025-10-27 11:04:51.458 INFO 2458 --- [host-check-clockdiff3,473decb77577a4a3,02676a3a8d7477db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534291 0 0 , error= 2025-10-27 11:04:51.511 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:04:51.511 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:04:51.511 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:04:51.513 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:04:51.512821659+08:00 2025-10-27 11:04:51.513 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:04:51.513444204+08:00 2025-10-27 11:04:51.514 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:04:51.514 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:04:51.513+08:00 2025-10-27 11:04:51.515 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:04:51.515 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:04:51.515 INFO 2458 --- [host-check-agent-status3,37714e3fa3dcf2d3,73a17aa2f50eeb38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:04:51.515 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:04:51.513+08:00 2025-10-27 11:04:51.516 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:04:51.516 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:04:51.516 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:04:51.517 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:04:51.517 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:04:51.517588607+08:00 2025-10-27 11:04:51.517 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:04:51.516868181+08:00 2025-10-27 11:04:51.518 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:04:51.517+08:00 2025-10-27 11:04:51.518 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:04:51.518 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:04:51.518 INFO 2458 --- [host-check-agent-status5,37714e3fa3dcf2d3,f57b0f07e49e60a7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:04:51.518 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:04:51.517+08:00 2025-10-27 11:04:51.519 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:04:51.519 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:04:51.519 INFO 2458 --- [host-check-agent-status1,37714e3fa3dcf2d3,0ecf65313a365e90] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:04:51.520 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:04:51.520 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:04:51.520 INFO 2458 --- [host-check-agent-status2,37714e3fa3dcf2d3,6779d514bbd5500a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:04:51.521 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:04:51.522 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:04:51.523 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:04:51.521941393+08:00 2025-10-27 11:04:51.523 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:04:51.523+08:00 2025-10-27 11:04:51.523 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:04:51.523 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:04:51.523 INFO 2458 --- [host-check-agent-status4,37714e3fa3dcf2d3,fcc3e7f9b6c86deb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:04:51.525 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:04:51.513+08:00 and status AVAILABLE 2025-10-27 11:04:51.525 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:04:51.513+08:00 and status AVAILABLE 2025-10-27 11:04:51.526 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:04:51.517+08:00 and status AVAILABLE 2025-10-27 11:04:51.526 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:04:51.517+08:00 and status AVAILABLE 2025-10-27 11:04:51.529 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:04:51.530 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:04:51.523+08:00 and status AVAILABLE 2025-10-27 11:04:51.532 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:04:51.533 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:04:51.533 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:04:51.535 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:04:51.545 INFO 2458 --- [ocp-schedules-21,53e8080055584887,47483009f87c4e76] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:04:51.549 INFO 2458 --- [ocp-schedules-5,e70fc5442e34a1f1,a8da4ca638007d0b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:04:51.550 INFO 2458 --- [ocp-schedules-21,53e8080055584887,47483009f87c4e76] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:51.551 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:04:51.552 INFO 2458 --- [ocp-schedules-5,e70fc5442e34a1f1,a8da4ca638007d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:51.553 INFO 2458 --- [ocp-schedules-5,e70fc5442e34a1f1,a8da4ca638007d0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:04:51.553 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:51.553 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:04:51.554 INFO 2458 --- [ocp-schedules-1,748ac618eca21dda,a9eebb1537ca9770] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:04:51.554 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:04:51.554 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:04:51.555 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:51.556 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:04:51.556 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:51.556 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:04:51.557 INFO 2458 --- [ocp-schedules-1,748ac618eca21dda,a9eebb1537ca9770] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:51.558 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:51.566 INFO 2458 --- [ocp-schedules-16,cfd0d337dcd0f87b,4c5ce0dc9ecda8f5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:04:51.568 INFO 2458 --- [ocp-schedules-2,0db977a10a8f11ff,8d553dcddaa3c45a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:04:51.569 INFO 2458 --- [ocp-schedules-10,02a7f8ead5118db6,6c0a8c9908a844b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:04:51.578 INFO 2458 --- [ocp-schedules-2,0db977a10a8f11ff,8d553dcddaa3c45a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:04:51.579 INFO 2458 --- [ocp-schedules-10,02a7f8ead5118db6,6c0a8c9908a844b7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:04:51.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:04:51.616 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:51.619 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.625 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:51.626 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.626 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:51.626 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:51.628 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:04:51.628 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.628 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:51.632 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:04:51.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:04:51.633 INFO 2458 --- [ocp-schedules-21,53e8080055584887,47483009f87c4e76] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:04:51.644 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:04:51.645 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:04:51.646 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:04:51.649 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:04:51.669 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:04:51.675 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:04:51.677 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:04:51.678 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:04:51.684 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:04:51.687 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.687 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:51.690 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:51.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:04:51.704 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:51.709 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:51.711 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:51.711 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:51.714 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:04:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:04:51.716175+08:00, lastProcessTime=2025-10-27T11:04:41.720051+08:00 2025-10-27 11:04:51.719 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:04:51.726 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:04:51.732 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:04:51.733 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:04:51.746 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:04:51.755 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:51.756 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.757 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:51.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.761 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:04:51.762 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:04:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:51.765 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:04:51.769 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:04:51.772 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:04:51.783 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:51.783 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.783 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.783 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:51.784 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:51.784 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:04:51.784 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.784 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.785 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.785 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:04:51.798 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:51.799 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:51.809 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:51.809 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:51.814 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:04:51.815 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:51.815 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:04:51.815 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:04:51.830 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:04:51.830 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:04:51.845 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:04:51.847 INFO 2458 --- [host-check-host-status5,0aa7be348725f200,0c309d4a4b4dbf36] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:04:51.852 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:51.853 INFO 2458 --- [host-check-host-status4,0aa7be348725f200,d6e28ccb84c4649b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:04:51.853 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.854 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:51.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:04:51.861 INFO 2458 --- [host-check-host-status3,0aa7be348725f200,93c36ccb389cb224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:04:51.861 INFO 2458 --- [host-check-host-status2,0aa7be348725f200,cca6e0d99c161726] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:04:51.865 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:04:51.876 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:04:51.879 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:04:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.882 INFO 2458 --- [host-check-host-status1,0aa7be348725f200,6e5a9825f010125d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:04:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:04:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:04:51.888 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:51.925 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:51.925 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:51.944 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:04:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:51.982 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:04:51.716175+08:00 2025-10-27 11:04:51.988 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:04:51.988 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.989 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:04:51.991 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:51.992 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:04:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:52.013 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:04:52.039 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:52.039 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:52.040 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:04:52.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:52.047 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:04:52.064 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:04:52.064 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:04:52.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:52.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:04:52.120 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.158 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:04:52.162 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:04:52.172 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:04:52.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:04:52.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:04:52.229 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:04:52.283 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:04:52.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.284 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:52.284 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:04:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:52.287 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:04:52.321 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:52.361 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:04:52.404 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:04:52.404 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:04:52.405 INFO 2458 --- [ocp-schedules-1,748ac618eca21dda,a9eebb1537ca9770] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:04:52.473 WARN 2458 --- [ocp-async-7,8f5a97746c59a2bf,5c382baaa0f6ae20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:52.473 INFO 2458 --- [ocp-async-3,842947d4cf10e39c,afde77636453db75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:04:53.376 INFO 2458 --- [ocp-schedules-8,0339ca335319648d,2397e289a2836648] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:04:53.380 INFO 2458 --- [ocp-schedules-8,0339ca335319648d,2397e289a2836648] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:04:53.707 INFO 2458 --- [prometheus-minute-schedule-1,3aea8158c7c2ec2f,2cefe098360d6acd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:53.714 INFO 2458 --- [prometheus-minute-schedule-1,3aea8158c7c2ec2f,2cefe098360d6acd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:53.933 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:53.934 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:54.285 INFO 2458 --- [ocp-schedules-16,7739d3a5eead8826,2c8aec64f383665e] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:04:54.473 WARN 2458 --- [ocp-async-3,47e0499e99588d9a,0ae296c68d68bd37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:54.473 INFO 2458 --- [ocp-async-10,db53014933da28d4,b07ca25eebef7057] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:55.713 INFO 2458 --- [ocp-schedules-9,13a48c6c9d74e6b6,13e28e89e2bde6f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:04:55.854 INFO 2458 --- [ocp-schedules-19,f02d24f97a2265cd,b3e673b5ff18641c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:04:55.863 INFO 2458 --- [ocp-schedules-19,f02d24f97a2265cd,b3e673b5ff18641c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:04:55.953 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:04:55.954 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:55.957 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:04:55.965 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:55.966 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:55.994 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:55.994 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:55.999 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:55.999 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.000 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:04:56.000 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.000 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.000 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.003 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:04:56.019 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.020 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:56.022 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.022 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.025 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.025 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.027 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.028 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:56.049 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.050 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.053 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.053 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.053 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.053 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:56.076 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.076 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.079 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.079 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.079 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.080 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:04:56.082 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.082 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.082 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.082 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.103 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.104 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.105 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.106 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.106 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.107 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.108 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.109 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.110 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.110 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.111 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.112 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.112 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.112 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.112 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.113 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:04:56.113 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.116 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:04:56.142 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.142 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.143 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.143 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:04:56.145 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.145 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.146 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.146 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:04:56.176 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:56.177 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.177 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.177 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.178 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.178 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:04:56.178 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:04:56.187 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.187 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:04:56.187 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.187 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:04:56.190 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:04:56.190 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:04:56.206 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.206 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.206 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.207 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:04:56.209 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.209 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.209 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.209 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:04:56.236 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.236 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.236 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.236 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:04:56.266 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.266 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:04:56.321 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.321 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.321 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.323 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.323 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.323 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.323 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.324 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.324 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.324 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.325 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.325 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.326 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.326 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.326 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.327 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.327 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.327 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.327 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.328 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.328 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.328 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.329 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.329 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.330 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.330 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.330 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.331 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.331 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.331 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.332 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.332 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.333 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.333 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.333 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.333 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.334 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.334 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.334 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.335 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:04:56.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.336 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:04:56.337 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.337 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.338 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:04:56.338 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:04:56.341 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:04:56.420 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:04:56.421 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:04:56.445 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:04:56.449 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:04:56.449 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:04:56.449 INFO 2458 --- [ocp-schedules-19,f02d24f97a2265cd,b3e673b5ff18641c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:04:56.473 WARN 2458 --- [ocp-async-10,6355acb95749abcc,61de6a688a351d4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:56.473 INFO 2458 --- [ocp-async-10,36846010722c6de0,31ba1318eaa71b88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:56.707 INFO 2458 --- [prometheus-minute-schedule-1,7890baa984c8bbb1,188aaf8022cd184c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:56.714 INFO 2458 --- [prometheus-minute-schedule-1,7890baa984c8bbb1,188aaf8022cd184c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:56.934 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:56.935 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:04:57.257 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,7ae83264e281b356,835779ff2c55c74f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=7ae83264e281b356, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:04:57.258 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,7ae83264e281b356,835779ff2c55c74f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=7ae83264e281b356, duration=1 ms] 2025-10-27 11:04:58.034 INFO 2458 --- [ocp-schedules-26,b82f9b958140d832,45611ecaea8750c1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:04:58.070 WARN 2458 --- [scanner-112126493-1,b82f9b958140d832,e630afb2c49d4e58] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:04:58.150 INFO 2458 --- [alarm-detect-3,b82f9b958140d832,2e5a8a8b8e8387e7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534270 ~ 1761534288 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534285, value=56.298)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534285, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534285=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:04:58.473 INFO 2458 --- [ocp-async-10,075e069c2d801e1d,f9e20e8d0713643b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:04:58.473 WARN 2458 --- [ocp-async-7,fa6b9958aaa06dc6,94c58158f6be5aa6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:04:58.507 INFO 2458 --- [ocp-async-3,e4544211be680bbe,42a1acba61fb069d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:04:59.122 INFO 2458 --- [ocp-schedules-15,980548ae9ca1fed9,eb94cd21ca6bfb9c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:04:59.707 INFO 2458 --- [prometheus-minute-schedule-1,d872e077d298eb16,59d5fdd1469d62bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534300, interval=60 2025-10-27 11:04:59.715 INFO 2458 --- [prometheus-minute-schedule-1,d872e077d298eb16,59d5fdd1469d62bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:04:59.935 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:04:59.936 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:00.338 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:00:00.0, nextRun=2025-10-27 11:05:00.0, currentDate=Mon Oct 27 11:05:00 CST 2025 2025-10-27 11:05:00.338 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:00:00.0, nextRunTime=2025-10-27 11:05:00.0} 2025-10-27 11:05:00.351 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:05:00.352 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:05:00.352+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:05:00.389 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:00:00.0, nextRun=2025-10-27 11:05:00.0, currentDate=Mon Oct 27 11:05:00 CST 2025 2025-10-27 11:05:00.389 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:00:00.0, nextRunTime=2025-10-27 11:05:00.0} 2025-10-27 11:05:00.449 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:05:00.457 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:05:00.464 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:05:00.464 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,b73a84c30a0ff588] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:05:00.464+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:05:00.473 INFO 2458 --- [ocp-async-3,99481699151a0b85,0c33363b9db86576] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:00.473 WARN 2458 --- [ocp-async-2,7ff5a51b4a826291,dd8a86d28151c31a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:00.541 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,d9e10405b5c1b217] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9144, operation:EXECUTE, state: RUNNING 2025-10-27 11:05:00.551 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,d9e10405b5c1b217] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9145, operation:EXECUTE, state: RUNNING 2025-10-27 11:05:00.569 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:05:00.570 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9144, context=Context{parallelIdx=-1, stringMap={task_instance_id=9094, task_operation=execute, latest_execution_start_time=2025-10-27T11:05:00.531+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9144}, listMap={}}, executor=10.40.65.146 2025-10-27 11:05:00.590 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:05:00.594 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9145, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9095, task_operation=execute, latest_execution_start_time=2025-10-27T11:05:00.549+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9145}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:05:00.594 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:05:00.633 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:00.753 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:05:00.756 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:05:00.757 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:00.823 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:00.838 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:05:00.857 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:05:00.883 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:00.993 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:05:01.057 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:01.198 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:01.250 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:05:01.250 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:05:01.398 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:05:01.427 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:05:01.436 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,caec0be881efc02c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9145, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:05:01.463 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:01.530 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:05:01.548 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:05:01.549 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:05:01.556 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.574 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.597 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.622 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.678 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.716 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:01.714955+08:00, lastProcessTime=2025-10-27T11:04:51.716175+08:00 2025-10-27 11:05:01.828 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.849 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:05:01.860 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:05:01.860 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:05:01.875 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,49d12637d6c2b292] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9144, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:05:01.878 INFO 2458 --- [ocp-schedules-24,5ee679e48b90d3f9,509642c714cdebff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:05:01.884 INFO 2458 --- [ocp-schedules-24,5ee679e48b90d3f9,509642c714cdebff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:01.924 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:01.925 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:05:01.939 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:01.714955+08:00 2025-10-27 11:05:01.943 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:01.943 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:01.943 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:01.943 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:01.943 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:01.944 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:05:01.958 INFO 2458 --- [ocp-schedules-24,5ee679e48b90d3f9,509642c714cdebff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:05:02.052 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9094, status:SUCCESS 2025-10-27 11:05:02.057 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9094, status:SUCCESS 2025-10-27 11:05:02.065 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9095, status:SUCCESS 2025-10-27 11:05:02.066 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9095, status:SUCCESS 2025-10-27 11:05:02.473 WARN 2458 --- [ocp-async-6,183502be16a94cd6,ff622fc64661b034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:02.473 INFO 2458 --- [ocp-async-1,e5bf3601a98b4564,41927bf961eab724] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:02.707 INFO 2458 --- [prometheus-minute-schedule-1,26c86e0420fc785b,a6b6dfbf14b2bc46] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:02.715 INFO 2458 --- [prometheus-minute-schedule-1,26c86e0420fc785b,a6b6dfbf14b2bc46] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:02.936 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:02.937 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:03.387 INFO 2458 --- [ocp-schedules-23,e24535879ca0af21,d53490b025a7ef10] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:03.391 INFO 2458 --- [ocp-schedules-23,e24535879ca0af21,d53490b025a7ef10] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:04.473 WARN 2458 --- [ocp-async-10,bed2275b57ffa65d,bc4e214909e7ca41] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:04.473 INFO 2458 --- [ocp-async-9,ed360d93e38b4bb9,56d406c6194a17de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:05.707 INFO 2458 --- [prometheus-minute-schedule-1,1663f8cb191d7188,f2d35611effaaa18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:05.715 INFO 2458 --- [prometheus-minute-schedule-1,1663f8cb191d7188,f2d35611effaaa18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:05.722 INFO 2458 --- [ocp-schedules-24,c4605f3078f27509,43c6a79eeed413bd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:05.937 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:05.937 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:06.473 WARN 2458 --- [ocp-async-2,8da9e6991b319cf7,55aadf08cf7cfca3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:06.473 INFO 2458 --- [ocp-async-4,8dfd36c2eaa1c8e2,a4fe4c9a5b0846cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:06.604 INFO 2458 --- [ocp-schedules-6,74ac51249e484494,520b3e5314e67888] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:05:06.611 INFO 2458 --- [ocp-schedules-6,74ac51249e484494,520b3e5314e67888] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:06.611 INFO 2458 --- [ocp-schedules-6,74ac51249e484494,520b3e5314e67888] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:08.156 INFO 2458 --- [ocp-schedules-3,b6ac6f74c379c671,db5fc83dcb773427] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:08.204 WARN 2458 --- [scanner-112126873-1,b6ac6f74c379c671,33056aff5282717e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:08.473 INFO 2458 --- [ocp-async-6,3d6ade5b2b69cb4b,3745a19669076aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:08.473 WARN 2458 --- [ocp-async-5,89268c75a1bd750b,61fc1b7befcd1fe4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:08.507 INFO 2458 --- [ocp-async-7,68f9599e329475d6,ec3d0e77f052e278] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:08.707 INFO 2458 --- [prometheus-minute-schedule-1,f27297864305a367,8f7ef21b3e320231] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:08.719 INFO 2458 --- [prometheus-minute-schedule-1,f27297864305a367,8f7ef21b3e320231] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:08.938 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:08.938 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:10.012 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:05:10.012 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:05:10.053 INFO 2458 --- [Collect obproxy1,a91e9a8575974c38,d00bbc3e9093a991] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:05:10.053 INFO 2458 --- [Check obproxy1,ce812f21018be60d,b6e1a2c1d05de59b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:05:10.053 INFO 2458 --- [Check obproxy1,ce812f21018be60d,b6e1a2c1d05de59b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:05:10.053 INFO 2458 --- [Collect obproxy1,a91e9a8575974c38,d00bbc3e9093a991] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:05:10.083 INFO 2458 --- [Collect obproxy1,a91e9a8575974c38,d00bbc3e9093a991] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:05:10.086 INFO 2458 --- [Check obproxy1,ce812f21018be60d,b6e1a2c1d05de59b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:05:10.086 INFO 2458 --- [Collect obproxy1,a91e9a8575974c38,d00bbc3e9093a991] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:05:10.090 INFO 2458 --- [Collect obproxy1,a91e9a8575974c38,d00bbc3e9093a991] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:05:10.090 INFO 2458 --- [Check obproxy1,ce812f21018be60d,b6e1a2c1d05de59b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:05:10.090 INFO 2458 --- [Check obproxy1,ce812f21018be60d,b6e1a2c1d05de59b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:05:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:05:10.474 INFO 2458 --- [ocp-async-7,1f65f6826dbbde64,1a47906a73f40dfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:10.474 WARN 2458 --- [ocp-async-9,d474a9b292b5cf56,eb5d65f8386a4b1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:11.707 INFO 2458 --- [prometheus-minute-schedule-1,a976f71e340c31fc,ad63e79cc0b7ed43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:11.716 INFO 2458 --- [prometheus-minute-schedule-1,a976f71e340c31fc,ad63e79cc0b7ed43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:11.738 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:11.737823+08:00, lastProcessTime=2025-10-27T11:05:01.714955+08:00 2025-10-27 11:05:11.938 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:11.939 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:11.943 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:11.737823+08:00 2025-10-27 11:05:12.474 INFO 2458 --- [ocp-async-3,bfa1f98769e48ec5,3c15176312469572] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:12.474 WARN 2458 --- [ocp-async-2,60b783792dc6900f,42f43c5ba635b57d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:13.399 INFO 2458 --- [ocp-schedules-11,67dd615d690116f2,7a919b9ab6c3e486] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:13.403 INFO 2458 --- [ocp-schedules-11,67dd615d690116f2,7a919b9ab6c3e486] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:13.508 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534240, nodeCount=821, affectRows=821 2025-10-27 11:05:13.693 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=1000, affectRows=1000 2025-10-27 11:05:13.696 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=994, validCount=992 2025-10-27 11:05:13.733 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=1000, affectRows=1000 2025-10-27 11:05:13.737 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=999, validCount=995 2025-10-27 11:05:13.750 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835014129065861120-1761534180' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835014129065861120-1761534180' for key 'PRIMARY' 2025-10-27 11:05:13.966 INFO 2458 --- [ocp-schedules-15,6aad04ac8db6ba3c,c63858be501907c1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:05:13.966 INFO 2458 --- [ocp-schedules-15,6aad04ac8db6ba3c,c63858be501907c1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:05:13.990 INFO 2458 --- [ocp-schedules-19,93d4b45f9016800c,18e9e1eaeaee71d0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:05:13.990 INFO 2458 --- [ocp-schedules-19,93d4b45f9016800c,18e9e1eaeaee71d0] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:05:13.997 INFO 2458 --- [ocp-schedules-19,93d4b45f9016800c,18e9e1eaeaee71d0] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:05:13.997 INFO 2458 --- [ocp-schedules-19,93d4b45f9016800c,18e9e1eaeaee71d0] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:05:14.026 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:05:14.027 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:05:14.027 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:05:14.027 INFO 2458 --- [Collect all tenants major compactions5,6aad04ac8db6ba3c,82961cb21ce95bbe] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:05:14.027 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:05:14.028 INFO 2458 --- [Collect all tenants major compactions6,6aad04ac8db6ba3c,17f183e72dbe86d4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:05:14.072 INFO 2458 --- [Collect all tenants major compactions6,6aad04ac8db6ba3c,17f183e72dbe86d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.074 INFO 2458 --- [Collect all tenants major compactions6,6aad04ac8db6ba3c,17f183e72dbe86d4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:05:14.074 INFO 2458 --- [Collect all tenants major compactions5,6aad04ac8db6ba3c,82961cb21ce95bbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.076 INFO 2458 --- [Collect all tenants major compactions5,6aad04ac8db6ba3c,82961cb21ce95bbe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:05:14.077 INFO 2458 --- [Collect all tenants major compactions6,6aad04ac8db6ba3c,17f183e72dbe86d4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:05:14.079 INFO 2458 --- [Collect all tenants major compactions5,6aad04ac8db6ba3c,82961cb21ce95bbe] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:05:14.092 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.092 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.093 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.093 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.093 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.094 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.096 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions3,6aad04ac8db6ba3c,517fc7a73f399186] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:14.098 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:05:14.099 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:05:14.101 INFO 2458 --- [Collect all tenants major compactions4,6aad04ac8db6ba3c,1eb279eda661090a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:05:14.101 INFO 2458 --- [Collect all tenants major compactions2,6aad04ac8db6ba3c,8a4b01fa9ec31934] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:05:14.103 INFO 2458 --- [Collect all tenants major compactions1,6aad04ac8db6ba3c,5b3b96b4a4b878c1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:05:14.104 INFO 2458 --- [ocp-schedules-15,6aad04ac8db6ba3c,c63858be501907c1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:05:14.156 INFO 2458 --- [ocp-schedules-26,2406e5a54b1a85f7,f8c83efda6a5379e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:05:14.156+08:00, clusters=[1, 2] 2025-10-27 11:05:14.160 INFO 2458 --- [ocp-schedules-26,2406e5a54b1a85f7,f8c83efda6a5379e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:05:14.164 INFO 2458 --- [ocp-schedules-26,2406e5a54b1a85f7,f8c83efda6a5379e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:05:14.366 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=1000, affectRows=1000 2025-10-27 11:05:14.371 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=996, validCount=992 2025-10-27 11:05:14.428 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=1000, affectRows=1000 2025-10-27 11:05:14.433 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=996, validCount=992 2025-10-27 11:05:14.450 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=74, affectRows=74 2025-10-27 11:05:14.451 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=74, validCount=74 2025-10-27 11:05:14.474 WARN 2458 --- [ocp-async-5,03490091b9b6d92e,f94cce01158cf703] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:14.474 INFO 2458 --- [ocp-async-1,a9ec2b4207e95b12,f6bb675bfbdb6d1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:14.707 INFO 2458 --- [prometheus-minute-schedule-1,502dd7ebb7d9731e,16f71e0945f7d28d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:14.715 INFO 2458 --- [prometheus-minute-schedule-1,502dd7ebb7d9731e,16f71e0945f7d28d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:14.939 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:14.940 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:15.727 INFO 2458 --- [ocp-schedules-18,2a33ae9a374aaa53,fa40d572a5f2e221] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:16.059 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,00550c2b79b01dc2,dbf61d3d75e0e767] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=00550c2b79b01dc2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:05:16.060 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,58c5657239034c86,4669c08c5de356c1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=58c5657239034c86, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:05:16.060 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,58c5657239034c86,4669c08c5de356c1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=58c5657239034c86, duration=6 ms] 2025-10-27 11:05:16.075 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6775a97fe4eb4f43,42f7d232879330a9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=6775a97fe4eb4f43, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:05:16.082 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7e4d8a5c1ea88afd,f1cd2a26dbb8d3dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=7e4d8a5c1ea88afd, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:05:16.084 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,00550c2b79b01dc2,dbf61d3d75e0e767] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=00550c2b79b01dc2, duration=33 ms] 2025-10-27 11:05:16.085 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7e4d8a5c1ea88afd,f1cd2a26dbb8d3dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=7e4d8a5c1ea88afd, duration=22 ms] 2025-10-27 11:05:16.190 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,ca1585daa9d5ea38,0d1ebffb9a170e8c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=ca1585daa9d5ea38, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:05:16.223 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,ca1585daa9d5ea38,0d1ebffb9a170e8c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=ca1585daa9d5ea38, duration=41 ms] 2025-10-27 11:05:16.319 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,0e3862a2891f70b0,20a812f982525410] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.40.198.13, traceId=0e3862a2891f70b0, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-10-27 11:05:16.326 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,89de0e6a78aca336,36e65e0c7036832f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.40.198.13, traceId=89de0e6a78aca336, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:05:16.343 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:16.343 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:16.343 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:16.358 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,0e3862a2891f70b0,20a812f982525410] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=0e3862a2891f70b0, duration=58 ms] 2025-10-27 11:05:16.384 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6775a97fe4eb4f43,42f7d232879330a9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=6775a97fe4eb4f43, duration=321 ms] 2025-10-27 11:05:16.413 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,89de0e6a78aca336,36e65e0c7036832f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.40.198.13, traceId=89de0e6a78aca336, duration=101 ms] 2025-10-27 11:05:16.474 WARN 2458 --- [ocp-async-10,49b029030e533f43,b8ecfaacf2785bc5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:16.474 INFO 2458 --- [ocp-async-8,421ed346f3e80bb0,57c794abac6aebf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:16.893 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,cf4791e61bb1d547,984c4394695864f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A05%3A17%2B08%3A00&endTime=2025-10-27T11%3A05%3A17%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=cf4791e61bb1d547, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:05:17+08:00,2025-10-27T11:05:17+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:05:16.894 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,99091ba0c73132eb,b5e5e934087a6c1e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A05%3A17%2B08%3A00&endTime=2025-10-27T11%3A05%3A17%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=99091ba0c73132eb, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:05:17+08:00,2025-10-27T11:05:17+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:05:16.894 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6b64d733fb3fde0f,b6327964c3ca3b2a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A05%3A17%2B08%3A00&endTime=2025-10-27T11%3A05%3A17%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=6b64d733fb3fde0f, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:05:17+08:00,2025-10-27T11:05:17+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:05:16.896 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,12b3377376fb4532,362e25a37afcfc87] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A05%3A17%2B08%3A00&endTime=2025-10-27T11%3A05%3A17%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=12b3377376fb4532, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:05:17+08:00,2025-10-27T11:05:17+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:05:16.998 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,12b3377376fb4532,362e25a37afcfc87] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=12b3377376fb4532, duration=115 ms] 2025-10-27 11:05:17.017 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,99091ba0c73132eb,b5e5e934087a6c1e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=99091ba0c73132eb, duration=134 ms] 2025-10-27 11:05:17.022 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6b64d733fb3fde0f,b6327964c3ca3b2a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=6b64d733fb3fde0f, duration=139 ms] 2025-10-27 11:05:17.045 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,cf4791e61bb1d547,984c4394695864f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=cf4791e61bb1d547, duration=162 ms] 2025-10-27 11:05:17.707 INFO 2458 --- [prometheus-minute-schedule-1,952372b1c4026277,cbdbd9da28817b4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:17.715 INFO 2458 --- [prometheus-minute-schedule-1,952372b1c4026277,cbdbd9da28817b4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:17.940 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:17.941 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:17.987 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,75027cab47ff22b6,d2d9c7630e710d24] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=75027cab47ff22b6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:05:17.987 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,75027cab47ff22b6,d2d9c7630e710d24] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=75027cab47ff22b6, duration=9 ms] 2025-10-27 11:05:17.987 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6a4b87078e844e85,9ca37241883f37a5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=6a4b87078e844e85, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:05:17.987 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,6a4b87078e844e85,9ca37241883f37a5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=6a4b87078e844e85, duration=9 ms] 2025-10-27 11:05:17.989 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,e76a3800e7d8c1cd,3c01017362d33fa9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=e76a3800e7d8c1cd, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:05:17.989 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,e76a3800e7d8c1cd,3c01017362d33fa9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=e76a3800e7d8c1cd, duration=10 ms] 2025-10-27 11:05:18.011 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534240, nodeCount=1000, affectRows=1000 2025-10-27 11:05:18.015 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534240, nodeCount=999, validCount=987 2025-10-27 11:05:18.017 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:18.017 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:18.017 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:18.357 INFO 2458 --- [ocp-schedules-3,0b4768e97aa1e54a,a8432cab78eefb53] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:18.390 WARN 2458 --- [scanner-112128617-1,0b4768e97aa1e54a,afea3db4eb347c18] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:18.474 WARN 2458 --- [ocp-async-7,d685b28811e6d987,ef95d0ccdc2e47ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:18.474 INFO 2458 --- [ocp-async-2,3d0eb9d135695cd8,4fa650c2327d959c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:18.476 INFO 2458 --- [alarm-detect-8,0b4768e97aa1e54a,255faa53855f021d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534285 ~ 1761534308 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=70.178)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534300=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:05:18.507 INFO 2458 --- [ocp-async-3,b64d9e44a0eae926,a21bd0e095ce7c73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:19.047 INFO 2458 --- [ocp-schedules-13,7680823ce7064b30,a1077b030e2eccab] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:05:19.193 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,45787614747c5c31,540d97b80326e05b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=45787614747c5c31, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:05:19.194 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,37eb4cd72fbeb801,7de5d38b468a0583] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=37eb4cd72fbeb801, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:05:19.194 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,37eb4cd72fbeb801,7de5d38b468a0583] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=37eb4cd72fbeb801, duration=13 ms] 2025-10-27 11:05:19.199 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1ecda1c6fbb6d9a7,29282e7d92c0181c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=1ecda1c6fbb6d9a7, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:05:19.200 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1ecda1c6fbb6d9a7,29282e7d92c0181c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=1ecda1c6fbb6d9a7, duration=13 ms] 2025-10-27 11:05:19.204 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,7ce6f8ee5c7689f4,beb92792748115dd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?null, client=10.40.198.13, traceId=7ce6f8ee5c7689f4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],NONE,] 2025-10-27 11:05:19.212 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,45787614747c5c31,540d97b80326e05b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=45787614747c5c31, duration=31 ms] 2025-10-27 11:05:19.221 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,7ce6f8ee5c7689f4,beb92792748115dd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.40.198.13, traceId=7ce6f8ee5c7689f4, duration=39 ms] 2025-10-27 11:05:19.226 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:19.226 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:19.226 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:19.269 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6770aa5f33ce45ac,f1fdf7ff175e31b4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=6770aa5f33ce45ac, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:05:19.270 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/summary?id=2&tenantId=6, client=10.40.198.13, traceId=bd9951966f45cf02, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantController.getTenantSummary(Long,Long), args=2,6,] 2025-10-27 11:05:19.278 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7e48cd9432412317,26920adc7a1c1e78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=2&tenantId=6&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=7e48cd9432412317, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,2,6,] 2025-10-27 11:05:19.310 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7e48cd9432412317,26920adc7a1c1e78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=7e48cd9432412317, duration=48 ms] 2025-10-27 11:05:19.359 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.360 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.364 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:05:19.371 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.372 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.376 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:05:19.378 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d8d04e772d65024a,bc94c55c2a7522d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=d8d04e772d65024a, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:05:19.382 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.386 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:05:19.402 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d8d04e772d65024a,bc94c55c2a7522d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=d8d04e772d65024a, duration=34 ms] 2025-10-27 11:05:19.403 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.403 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.472 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.473 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.529 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6770aa5f33ce45ac,f1fdf7ff175e31b4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=6770aa5f33ce45ac, duration=267 ms] 2025-10-27 11:05:19.538 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.539 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.543 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.550 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.551 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:05:19.620 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.621 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2025-10-27 11:05:19.622 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.624 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.625 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:05:19.625 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [ob_enable_sql_audit] 2025-10-27 11:05:19.662 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=72cf22ebc52214ee, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:05:19.662 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b824d9b0c0486e32,c3e6ad9ea8c76269] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=10.40.198.13, traceId=b824d9b0c0486e32, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2025-10-27 11:05:19.668 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants?id=2&tenantId=6, client=10.40.198.13, traceId=9e31df2beccee82e, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantStandbyController.listRelationTenants(Long,Long), args=2,6,] 2025-10-27 11:05:19.698 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.704 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.706 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:05:19.713 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b824d9b0c0486e32,c3e6ad9ea8c76269] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=10.40.198.13, traceId=b824d9b0c0486e32, duration=60 ms] 2025-10-27 11:05:19.718 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.718 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.722 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:05:19.730 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.730 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.736 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:05:19.740 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.743 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.744 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:05:19.744 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:05:19.747 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.747 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.749 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:05:19.750 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.750 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.760 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:05:19.760 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bd9951966f45cf02,113cace4809d642e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/summary client=10.40.198.13, traceId=bd9951966f45cf02, duration=498 ms] 2025-10-27 11:05:19.768 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.770 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:05:19.777 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.777 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:05:19.780 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.781 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:05:19.785 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.786 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [resource_hard_limit] 2025-10-27 11:05:19.801 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.806 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2025-10-27 11:05:19.819 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,72cf22ebc52214ee,7bf3fcc528cfcf33] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=72cf22ebc52214ee, duration=166 ms] 2025-10-27 11:05:19.824 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.824 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.872 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=9fc8babd36cc79d0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:05:19.873 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d39869b6a4a6a124,adcf2291feeede0a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=10.40.198.13, traceId=d39869b6a4a6a124, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2025-10-27 11:05:19.875 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7d830da93d16bc2f,76052c900a9a4946] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/binlogService?id=2&tenantId=6, client=10.40.198.13, traceId=7d830da93d16bc2f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.getTenantBinlogService(Long,Long), args=2,6,] 2025-10-27 11:05:19.876 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7d830da93d16bc2f,76052c900a9a4946] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/binlogService client=10.40.198.13, traceId=7d830da93d16bc2f, duration=14 ms] 2025-10-27 11:05:19.884 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.884 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.890 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2025-10-27 11:05:19.898 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.899 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:19.906 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.907 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.908 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:05:19.923 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d39869b6a4a6a124,adcf2291feeede0a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=10.40.198.13, traceId=d39869b6a4a6a124, duration=61 ms] 2025-10-27 11:05:19.930 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.933 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:05:19.949 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.950 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:05:19.950 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:19.951 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.952 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:05:19.956 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:05:19.956 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.o.i.t.c.TenantOperatorHelperImpl : These operators for clusterIds=[] have been created. 2025-10-27 11:05:19.956 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,9e31df2beccee82e,61c46c8ea5d1062d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants client=10.40.198.13, traceId=9e31df2beccee82e, duration=303 ms] 2025-10-27 11:05:19.970 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.972 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:05:19.974 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.974 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:05:19.975 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:19.976 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:05:19.998 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants?id=2&tenantId=6, client=10.40.198.13, traceId=5d6f7313099e4fcd, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantStandbyController.listRelationTenants(Long,Long), args=2,6,] 2025-10-27 11:05:20.011 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9fc8babd36cc79d0,03f7eeafc5473d1c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=9fc8babd36cc79d0, duration=149 ms] 2025-10-27 11:05:20.039 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:20.040 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.045 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:05:20.052 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:20.053 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.058 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:05:20.062 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.066 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:05:20.072 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.072 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:20.135 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.136 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:20.200 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.200 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:20.204 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:20.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.206 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:05:20.229 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.231 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:05:20.249 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:20.250 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:20.251 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:05:20.255 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:05:20.255 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.o.i.t.c.TenantOperatorHelperImpl : These operators for clusterIds=[] have been created. 2025-10-27 11:05:20.255 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5d6f7313099e4fcd,a83b8e89be1de18f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants client=10.40.198.13, traceId=5d6f7313099e4fcd, duration=264 ms] 2025-10-27 11:05:20.268 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,85039650592c1dd3,4688b905f166f146] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=85039650592c1dd3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:05:20.276 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,85039650592c1dd3,4688b905f166f146] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=85039650592c1dd3, duration=18 ms] 2025-10-27 11:05:20.474 WARN 2458 --- [ocp-async-6,0d0b2c75addd6d8f,8171b2fc08941c1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:20.474 INFO 2458 --- [ocp-async-5,cb5be2917b37808b,3d894aa3be26b3f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:20.707 INFO 2458 --- [prometheus-minute-schedule-1,9f471777ece09bf5,55cdc63daf553963] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:20.713 INFO 2458 --- [prometheus-minute-schedule-1,9f471777ece09bf5,55cdc63daf553963] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:20.941 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:20.942 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:20.988 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,bbb9bf755069267d,dcd46c81a137e206] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=bbb9bf755069267d, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:05:21.014 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,bbb9bf755069267d,dcd46c81a137e206] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=bbb9bf755069267d, duration=35 ms] 2025-10-27 11:05:21.174 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,9bdc6d72a84de2c2,23a60dace267d0a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=9bdc6d72a84de2c2, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:05:21.174 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,9bdc6d72a84de2c2,23a60dace267d0a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=9bdc6d72a84de2c2, duration=8 ms] 2025-10-27 11:05:21.182 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b08d1ffd073c66ec,1f83436880613a40] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=b08d1ffd073c66ec, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:05:21.182 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b08d1ffd073c66ec,1f83436880613a40] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=b08d1ffd073c66ec, duration=16 ms] 2025-10-27 11:05:21.183 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,3d2e9b4d0b24fd5f,a09755afc4cc343b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=3d2e9b4d0b24fd5f, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:05:21.183 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,3d2e9b4d0b24fd5f,a09755afc4cc343b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=3d2e9b4d0b24fd5f, duration=17 ms] 2025-10-27 11:05:21.212 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:21.212 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:21.212 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:21.409 INFO 2458 --- [ocp-schedules-30,ba612a958d4012ba,e523199302cfa1d5] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:21.410 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:05:21.410 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:05:21.410 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:05:21.410 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:05:21.410 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:05:21.411 INFO 2458 --- [ocp-schedules-14,7841f81976e7c36a,338cd09faafe76d7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:21.411 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:05:21.411 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:05:21.412 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:05:21.412 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:05:21.412 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:05:21.426 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:21.427 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:21.427 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:21.428 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:21.429 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:21.429 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:21.429 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:21.431 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:21.433 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:21.433 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:21.433 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:21.433 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:21.433 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:21.434 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:21.434 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:21.434 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:21.436 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:21.436 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:21.441 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:21.441 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:21.441 INFO 2458 --- [ocp-schedules-4,5fe8ce96a5a3e3c3,c16ba6e8364f5f68] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:21.442 INFO 2458 --- [ocp-schedules-16,9285a58e13f2360c,0992ece297d03972] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:05:21.442 INFO 2458 --- [host-check-clockdiff2,5fe8ce96a5a3e3c3,2b09b04ba9986672] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:05:21.442 INFO 2458 --- [host-check-clockdiff3,5fe8ce96a5a3e3c3,9b3e50e2177c1285] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:05:21.442 INFO 2458 --- [host-check-clockdiff1,5fe8ce96a5a3e3c3,8e4da273336caf1b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:05:21.442 INFO 2458 --- [host-check-clockdiff4,5fe8ce96a5a3e3c3,68ea102148faa006] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:05:21.442 INFO 2458 --- [host-check-clockdiff5,5fe8ce96a5a3e3c3,037f5a33fe906970] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:05:21.478 INFO 2458 --- [host-check-clockdiff2,5fe8ce96a5a3e3c3,2b09b04ba9986672] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534321 -1 -1 , error= 2025-10-27 11:05:21.476 INFO 2458 --- [host-check-clockdiff3,5fe8ce96a5a3e3c3,9b3e50e2177c1285] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534321 -1 -1 , error= 2025-10-27 11:05:21.490 INFO 2458 --- [host-check-clockdiff5,5fe8ce96a5a3e3c3,037f5a33fe906970] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534321 0 0 , error= 2025-10-27 11:05:21.491 INFO 2458 --- [host-check-clockdiff1,5fe8ce96a5a3e3c3,8e4da273336caf1b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534321 0 0 , error= 2025-10-27 11:05:21.507 INFO 2458 --- [host-check-clockdiff4,5fe8ce96a5a3e3c3,68ea102148faa006] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534321 0 0 , error= 2025-10-27 11:05:21.528 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:05:21.528 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:05:21.530 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:05:21.529517767+08:00 2025-10-27 11:05:21.530 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:05:21.530 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:05:21.530+08:00 2025-10-27 11:05:21.532 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:05:21.532 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:05:21.532 INFO 2458 --- [host-check-agent-status5,ba612a958d4012ba,d94741b5001d81b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:05:21.534 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:05:21.534 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:05:21.534 INFO 2458 --- [host-check-agent-status2,ba612a958d4012ba,c93d3998e2d1b463] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:05:21.539 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:05:21.540 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:05:21.541 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:05:21.540613331+08:00 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:05:21.541+08:00 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-agent-status3,ba612a958d4012ba,824b3d05ef4c2de6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:05:21.542 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:05:21.543 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:05:21.541778972+08:00 2025-10-27 11:05:21.543 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:05:21.543 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:05:21.543535052+08:00 2025-10-27 11:05:21.543 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:05:21.543 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:05:21.543+08:00 2025-10-27 11:05:21.544 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:05:21.543+08:00 2025-10-27 11:05:21.546 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:05:21.546 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:05:21.546 INFO 2458 --- [host-check-agent-status4,ba612a958d4012ba,6687ebb49b5addd7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:05:21.546 INFO 2458 --- [ocp-schedules-25,e75c398aed78713f,6f6caec2a0a21670] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:05:21.546 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:05:21.530+08:00 and status AVAILABLE 2025-10-27 11:05:21.547 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:05:21.548 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:05:21.548 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:05:21.548 INFO 2458 --- [host-check-agent-status1,ba612a958d4012ba,c524df76ad236f71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:05:21.548 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:05:21.547873644+08:00 2025-10-27 11:05:21.548 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:05:21.548+08:00 2025-10-27 11:05:21.549 INFO 2458 --- [ocp-schedules-8,0aba1f854d8a28d1,ded7ae6356dd775f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:05:21.551 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:05:21.543+08:00 and status AVAILABLE 2025-10-27 11:05:21.552 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:05:21.541+08:00 and status AVAILABLE 2025-10-27 11:05:21.553 INFO 2458 --- [ocp-schedules-8,0aba1f854d8a28d1,ded7ae6356dd775f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:21.553 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:05:21.553 INFO 2458 --- [ocp-schedules-25,e75c398aed78713f,6f6caec2a0a21670] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:21.554 INFO 2458 --- [ocp-schedules-17,595fbea7f8e7b797,76a6a29d17c9eff0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:05:21.556 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:05:21.556 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:05:21.548+08:00 and status AVAILABLE 2025-10-27 11:05:21.557 INFO 2458 --- [ocp-schedules-17,595fbea7f8e7b797,76a6a29d17c9eff0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:21.558 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:05:21.559 INFO 2458 --- [ocp-schedules-8,0aba1f854d8a28d1,ded7ae6356dd775f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:05:21.562 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:05:21.568 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:05:21.543+08:00 and status AVAILABLE 2025-10-27 11:05:21.569 INFO 2458 --- [ocp-schedules-23,aec28e05d745f4d0,cc80faa581bac7dd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:05:21.572 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:05:21.578 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:21.580 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:21.586 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:21.587 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:21.589 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:21.590 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:21.602 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:21.602 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:21.605 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:21.605 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:21.608 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.610 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:21.611 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:05:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.629 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:21.634 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.635 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:21.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:05:21.635 INFO 2458 --- [ocp-schedules-25,e75c398aed78713f,6f6caec2a0a21670] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:05:21.637 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:21.653 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:05:21.656 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:21.661 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:21.665 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:21.670 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:21.678 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:05:21.679 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:05:21.683 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.683 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:21.691 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:05:21.696 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:05:21.698 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:21.698 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:05:21.699 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.699 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:05:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:21.714922+08:00, lastProcessTime=2025-10-27T11:05:11.737823+08:00 2025-10-27 11:05:21.726 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:05:21.727 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:05:21.729 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:05:21.737 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:05:21.742 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:21.757 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:05:21.761 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:21.763 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:21.763 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:21.766 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:21.768 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:21.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.771 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:05:21.772 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:21.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:21.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.777 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.777 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.777 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.777 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:05:21.781 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:05:21.783 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:05:21.785 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:05:21.793 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:05:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:21.801 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:05:21.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:21.825 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:05:21.827 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:05:21.827 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:05:21.828 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:21.836 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:05:21.843 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:05:21.850 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:21.851 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.851 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:05:21.855 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:21.861 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:21.864 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:05:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.868 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:21.868 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:05:21.874 INFO 2458 --- [host-check-host-status4,7841f81976e7c36a,f9032aaf135955aa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:05:21.874 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:21.875 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:05:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:21.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:21.881 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:05:21.885 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:05:21.907 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:05:21.910 INFO 2458 --- [host-check-host-status2,7841f81976e7c36a,f82e2a185884b1a8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:05:21.917 INFO 2458 --- [host-check-host-status3,7841f81976e7c36a,349816b7ce9253a7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:05:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:21.923 INFO 2458 --- [host-check-host-status5,7841f81976e7c36a,194d039af69eb99e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:05:21.935 INFO 2458 --- [host-check-host-status1,7841f81976e7c36a,26a93216e7386de0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:05:21.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:05:21.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:21.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:21.985 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:21.714922+08:00 2025-10-27 11:05:21.990 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:05:21.990 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.991 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:05:21.993 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:05:22.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:22.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:22.011 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:05:22.034 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:22.035 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:22.036 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:05:22.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:22.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:22.058 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:05:22.058 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:05:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:22.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:22.163 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:05:22.178 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:05:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:22.194 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:05:22.296 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:05:22.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.296 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:22.297 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:05:22.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.299 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.299 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:22.300 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:05:22.333 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:05:22.371 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:22.371 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.371 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.371 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:22.371 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:22.372 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:22.372 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:05:22.419 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:05:22.419 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:05:22.419 INFO 2458 --- [ocp-schedules-17,595fbea7f8e7b797,76a6a29d17c9eff0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:05:22.475 WARN 2458 --- [ocp-async-10,611f95099d3e3bd5,dfc1a97f9b8ee02e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:22.475 INFO 2458 --- [ocp-async-9,38d47dc0e485d100,883243c1a53f9564] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:23.413 INFO 2458 --- [ocp-schedules-2,214edd3ba386e1e1,7d234efd711241d3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:23.416 INFO 2458 --- [ocp-schedules-2,214edd3ba386e1e1,7d234efd711241d3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:23.707 INFO 2458 --- [prometheus-minute-schedule-1,3b0df42ed6cc1546,0d3114b6e35981c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:23.714 INFO 2458 --- [prometheus-minute-schedule-1,3b0df42ed6cc1546,0d3114b6e35981c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:23.942 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:23.943 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:24.156 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,76bc91298b148daa,076a754a454d9919] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=76bc91298b148daa, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:05:24.159 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,76bc91298b148daa,076a754a454d9919] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=76bc91298b148daa, duration=12 ms] 2025-10-27 11:05:24.475 WARN 2458 --- [ocp-async-7,6520d57e91025461,89b3984e29ab52cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:24.475 INFO 2458 --- [ocp-async-4,aab6366df6dc9d9b,bd14b8fe7f156895] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:25.168 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:05:25.168 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:05:25.172 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:05:25.172 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:05:25.172 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:05:25.172 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:05:25.176 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:05:25.176 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:05:25.176 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:05:25.176 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:05:25.201 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:05:25.207 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:05:25.207 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:05:25.207 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:05:25.207 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:05:25.207 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:05:25.214 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:05:25.585 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:05:25.735 INFO 2458 --- [ocp-schedules-27,6ae618f65e61bd41,d5bdeffa64809e6f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:25.983 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.807101009S 2025-10-27 11:05:25.983 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:05:26.062 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,37455e05348a507c,f097fa405f3c8d74] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters?null, client=10.40.198.13, traceId=37455e05348a507c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClusters(Pageable), args=Page request [number: 0, size 5000, sort: name: ASC],] 2025-10-27 11:05:26.063 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,406a8161b02c5143,deba4a2c2eb9ecc8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/obproxy/clusters?null, client=10.40.198.13, traceId=406a8161b02c5143, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObproxyController.listObproxyClusters(Pageable,List,String,List,Boolean), args=Page request [number: 0, size 5000, sort: id: ASC],false,] 2025-10-27 11:05:26.090 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:05:26.090 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:05:26.094 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,37455e05348a507c,f097fa405f3c8d74] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters client=10.40.198.13, traceId=37455e05348a507c, duration=44 ms] 2025-10-27 11:05:26.121 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,406a8161b02c5143,deba4a2c2eb9ecc8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/obproxy/clusters client=10.40.198.13, traceId=406a8161b02c5143, duration=71 ms] 2025-10-27 11:05:26.136 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:05:26.181 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:05:26.190 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:05:26.305 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] o.s.boot.SpringApplication : Started application in 1.133 seconds (JVM running for 256794.732) 2025-10-27 11:05:26.305 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.133187835S 2025-10-27 11:05:26.475 WARN 2458 --- [ocp-async-3,45020e97ccb7ae9c,dcd725c314e8b859] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:26.475 INFO 2458 --- [ocp-async-6,5edffb134843ee1e,14d4f803ab3563c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:26.614 INFO 2458 --- [ocp-schedules-30,bb0b6cfa744fcc7e,ba0511d8edbfb999] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:05:26.621 INFO 2458 --- [ocp-schedules-30,bb0b6cfa744fcc7e,ba0511d8edbfb999] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:26.621 INFO 2458 --- [ocp-schedules-30,bb0b6cfa744fcc7e,ba0511d8edbfb999] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:26.696 INFO 2458 --- [ocp-schedules-14,df72e30be9703c1f,06afb3b69d0d1f6a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:05:26.707 INFO 2458 --- [prometheus-minute-schedule-1,fef88458d3b54fda,5ca61e1b8fdd8061] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:26.714 INFO 2458 --- [prometheus-minute-schedule-1,fef88458d3b54fda,5ca61e1b8fdd8061] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:26.943 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:26.943 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:28.475 WARN 2458 --- [ocp-async-10,d3a7d0aa30320dc6,5b947e8064db1c13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:28.475 INFO 2458 --- [ocp-async-8,89c9c47f1af460bc,3d40bdee92470dfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:28.507 INFO 2458 --- [ocp-async-9,770f0399282c287a,a5a0eb82fc0db255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:28.516 INFO 2458 --- [ocp-schedules-5,4c2e5b8c387358b2,8b6ff4e7d6edb6db] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:28.556 WARN 2458 --- [scanner-112128824-1,4c2e5b8c387358b2,a027d0c6753ddb55] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:28.690 INFO 2458 --- [alarm-detect-1,4c2e5b8c387358b2,9b5ec331391d104a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534300 ~ 1761534318 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534315, value=75.216)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534315, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534315=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:05:28.818 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:05:28.818 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:05:28.826 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:05:28.826 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:05:28.833 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:05:28.833 INFO 2458 --- [ocp-schedules-30,52c4f5b3da07a541,63e3345d112c26e7] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:05:29.295 INFO 2458 --- [ocp-schedules-3,2e924df9d54fe55b,dcff82399a2d0129] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:05:29.302 INFO 2458 --- [ocp-schedules-3,2e924df9d54fe55b,dcff82399a2d0129] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:05:29.302 INFO 2458 --- [ocp-schedules-3,2e924df9d54fe55b,dcff82399a2d0129] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:05:29.707 INFO 2458 --- [prometheus-minute-schedule-1,1a369925574ee54f,142a18795a388241] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:29.715 INFO 2458 --- [prometheus-minute-schedule-1,1a369925574ee54f,142a18795a388241] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:29.943 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:29.944 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:30.475 WARN 2458 --- [ocp-async-7,2c2bbb3bbb216809,50e5cacb7d3d35e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:30.475 INFO 2458 --- [ocp-async-2,88e17bc3a658e862,ee2bb579b366c006] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:31.714067+08:00, lastProcessTime=2025-10-27T11:05:21.714922+08:00 2025-10-27 11:05:31.922 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:31.714067+08:00 2025-10-27 11:05:32.475 WARN 2458 --- [ocp-async-2,998b1ada58416549,0c926184b322b90a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:32.475 INFO 2458 --- [ocp-async-2,fdbd2de76e417aab,54c7bcee03bda1b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:32.707 INFO 2458 --- [prometheus-minute-schedule-1,d58e54c7298780f9,68dc117be2c78c2c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:32.715 INFO 2458 --- [prometheus-minute-schedule-1,d58e54c7298780f9,68dc117be2c78c2c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:32.944 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:32.945 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:33.424 INFO 2458 --- [ocp-schedules-24,44b05a689ce6afcc,8fc174346003c701] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:33.427 INFO 2458 --- [ocp-schedules-24,44b05a689ce6afcc,8fc174346003c701] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:33.717 INFO 2458 --- [ocp-schedules-28,ae9926b272b96930,54ccd4eb034449e7] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:05:33.717 INFO 2458 --- [ocp-schedules-28,ae9926b272b96930,54ccd4eb034449e7] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:05:34.475 WARN 2458 --- [ocp-async-2,e058e0cf30116451,482a6481d1844787] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:34.475 INFO 2458 --- [ocp-async-2,59c179747f72e566,b371825a9bce87fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:34.501 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:05:34.501 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:05:34.566 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:34.566 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:34.598 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/ob/clusters/2/tenants/6/binlogService/activate, client 10.40.198.13, traceId c791b4f95d069192] 2025-10-27 11:05:34.617 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:34.617 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:34.617 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:34.619 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:05:34.634 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:34.634 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:34.634 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/ob/clusters/2/tenants/6/binlogService/activate?id=2&tenantId=6, client=10.40.198.13, traceId=c791b4f95d069192, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.activateTenantBinlogService(Long,Long,ActivateTenantBinlogServiceRequest), args=2,6,ActivateTenantBinlogServiceRequest(binlogClusterId=4, obproxyClusterIdList=[1], binlogFileReserveHours=1, binlogFileDiskSizeLimit=1, instanceCount=1),] 2025-10-27 11:05:34.634 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:34.635 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:05:34.706 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:34.707 INFO 2458 --- [ocp-schedules-10,2e23ca85f46c0b1f,ad53b2c89abc0403] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:05:34.708 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Create tenant binlog service, nodeCount=9, taskCount=9 2025-10-27 11:05:34.708 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Create tenant binlog service', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=6, hostId=null, creator='admin', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={binlog_instance_replicate_count=1, tenant_id=6, tenant_binlog_service_id=9, operation_scope=Tenant, prohibit_rollback=false, binlog_cluster_id=4}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, createTime=null, startTime=2025-10-27T11:05:34.708+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Prepare create tenant binlog service, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.PrepareCreateTenantBinlogServiceTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Create cluster cdcro user, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.CreateClusterCdcroUserTask, seriesId=6, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update cdcro user info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.UpdateClusterCdcroUserInfoTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Create tenant binlog service, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.CreateTenantBinlogServiceTask, seriesId=7, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Update tenant binlog service status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.UpdateTenantBinlogServiceStatusWhenCreateTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Set binlog instance parameter, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.SetBinlogInstanceParameterTask, seriesId=3, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null 2025-10-27 11:05:34.748 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/ob/clusters/2/tenants/6/binlogService/activate client=10.40.198.13, traceId=c791b4f95d069192, duration=157 ms] 2025-10-27 11:05:34.749 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,c791b4f95d069192,1d15bd4a82eb1959] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=ACTIVATE_TENANT_BINLOG_SERVICE, eventType=OB_TENANT_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.activateTenantBinlogService, resourceType=OB_TENANT, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=6, resourceName=test2, parentResource=myoceanbase2:1760510188)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=9096, eventInfo={"request":{"binlogClusterId":4,"binlogFileDiskSizeLimit":1,"binlogFileReserveHours":1,"instanceCount":1,"obproxyClusterIdList":[1]},"tenantId":6,"id":2}, executionType=NEW, requestUrl=http://10.40.198.13:24631/api/v2/ob/clusters/2/tenants/6/binlogService/activate, clientIp=10.40.198.13, serverIp=10.40.65.146, startTime=2025-10-27T11:05:34.634+08:00, endTime=null, role=null, aliUid=null) 2025-10-27 11:05:34.943 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/binlogService?id=2&tenantId=6, client=10.40.198.13, traceId=07577d8033c1eba5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.getTenantBinlogService(Long,Long), args=2,6,] 2025-10-27 11:05:34.944 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,fabe63aa4d771050,1cbda45171ce3e98] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=fabe63aa4d771050, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:05:34.945 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,27f14c529d18b674,c6f48fa878f2942d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=2&tenantId=6&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=27f14c529d18b674, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,2,6,] 2025-10-27 11:05:34.945 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,fabe63aa4d771050,1cbda45171ce3e98] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=fabe63aa4d771050, duration=10 ms] 2025-10-27 11:05:34.948 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,27f14c529d18b674,c6f48fa878f2942d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=27f14c529d18b674, duration=13 ms] 2025-10-27 11:05:34.992 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:05:34.992 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW VARIABLES FOR INSTANCE ?, args: [null] 2025-10-27 11:05:35.162 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] query failed, sql: [SHOW VARIABLES FOR INSTANCE ?], error message: [PreparedStatementCallback; SQL [SHOW VARIABLES FOR INSTANCE ?]; (conn=7896) Connection is closed; nested exception is java.sql.SQLNonTransientConnectionException: (conn=7896) Connection is closed] 2025-10-27 11:05:35.164 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.b.i.TenantBinlogManagerImpl : Can't get info from logproxy process, exception: com.oceanbase.ocp.binlogsdk.operator.BinlogExecuteException: (conn=7896) Connection is closed at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.customException(BinlogTemplate.java:121) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.customException(BinlogTemplate.java:113) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.query(BinlogTemplate.java:43) at com.oceanbase.ocp.binlogsdk.operator.JdbcBinlogOperator.listInstanceVariables(JdbcBinlogOperator.java:168) at com.oceanbase.ocp.binlog.internal.TenantBinlogManagerImpl.attachParameters(TenantBinlogManagerImpl.java:120) at com.oceanbase.ocp.binlog.internal.TenantBinlogManagerImpl.getTenantBinlogService(TenantBinlogManagerImpl.java:109) at com.oceanbase.ocp.binlog.internal.TenantBinlogManagerImpl$$FastClassBySpringCGLIB$$21866b48.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.binlog.internal.TenantBinlogManagerImpl$$EnhancerBySpringCGLIB$$eb264895.getTenantBinlogService() at com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.getTenantBinlogService(ObTenantBinlogController.java:52) at com.oceanbase.ocp.server.common.controller.ObTenantBinlogController$$FastClassBySpringCGLIB$$12011572.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:135) at sun.reflect.GeneratedMethodAccessor340.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.ObTenantBinlogController$$EnhancerBySpringCGLIB$$24eff2d6.getTenantBinlogService() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898) at javax.servlet.http.HttpServlet.service(HttpServlet.java:529) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:117) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLNonTransientConnectionException: (conn=7896) Connection is closed at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:300) at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:378) at org.mariadb.jdbc.client.impl.StandardClient.checkNotClosed(StandardClient.java:1404) at org.mariadb.jdbc.client.impl.StandardClient.sendQuery(StandardClient.java:973) at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:1144) at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:87) at org.mariadb.jdbc.ClientPreparedStatement.executeQuery(ClientPreparedStatement.java:252) at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757) at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.query(BinlogTemplate.java:40) ... 172 common frames omitted 2025-10-27 11:05:35.164 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,07577d8033c1eba5,82991bf97361de5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/binlogService client=10.40.198.13, traceId=07577d8033c1eba5, duration=229 ms] 2025-10-27 11:05:35.701 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,8bc1360056bccef7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9146, operation:EXECUTE, state: RUNNING 2025-10-27 11:05:35.707 INFO 2458 --- [prometheus-minute-schedule-1,de2b03132bff711a,28eea891cc7ffdf7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:35.714 INFO 2458 --- [prometheus-minute-schedule-1,de2b03132bff711a,28eea891cc7ffdf7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:35.744 INFO 2458 --- [ocp-schedules-14,ab44798d241d3902,a50569737e8c7fc3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:35.740 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,57ea1e74a96383ac] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:05:35.746 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,57ea1e74a96383ac] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9146, context=Context{parallelIdx=-1, stringMap={binlog_instance_replicate_count=1, tenant_id=6, tenant_binlog_service_id=9, operation_scope=Tenant, task_instance_id=9096, task_operation=execute, latest_execution_start_time=2025-10-27T11:05:35.695+08:00, sub_task_instance_name=Prepare create tenant binlog service, sub_task_instance_id=9146, binlog_cluster_id=4}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, executor=10.40.65.146 2025-10-27 11:05:35.758 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,57ea1e74a96383ac] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9146, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:05:35.766 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,57ea1e74a96383ac] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9147] 2025-10-27 11:05:35.945 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:35.946 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:36.476 INFO 2458 --- [ocp-async-2,53bcf4da883c0590,19588d24bf01afa0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:36.476 WARN 2458 --- [ocp-async-9,d13750e2e742eda4,a110397e04dcbcfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:36.719 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,e413543c1adbcf6c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9147, operation:EXECUTE, state: RUNNING 2025-10-27 11:05:36.748 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:36.748 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:05:36.748 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:05:36.748 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:05:36.749 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:05:36.749 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:05:36.749 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:05:36.763 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:36.763 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:36.766 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:36.767 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:36.767 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:36.767 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:36.758 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9147, context=Context{parallelIdx=-1, stringMap={binlog_instance_replicate_count=1, tenant_id=6, tenant_binlog_service_id=9, target_tenant_binlog_service_status=RUNNING, operation_scope=Tenant, task_instance_id=9096, task_operation=execute, latest_execution_start_time=2025-10-27T11:05:36.717+08:00, sub_task_instance_name=Create cluster cdcro user, sub_task_instance_id=9147, binlog_cluster_id=4}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, executor=10.40.65.146 2025-10-27 11:05:36.770 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:36.770 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:36.771 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:36.775 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:36.839 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:05:36.846 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?backUrl=/task, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:36.846 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/task/9096, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:36.846 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:36.849 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:05:36.852 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:05:36.852 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:05:36.852 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:05:36.854 INFO 2458 --- [ocp-schedules-26,52d1c30fa32f43a0,de6568f029b7f50d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:05:36.917 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:36.925 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `user`, (CASE `account_locked` WHEN 'Y' THEN 1 ELSE 0 END) AS account_locked, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `process_priv`, `grant_priv`, `index_priv`, `alter_priv`, `show_db_priv`, `super_priv`, `create_view_priv`, `show_view_priv`, `create_user_priv`, `create_routine_priv`, `alter_routine_priv`, `execute_priv`, `password`, `host` FROM `mysql`.`user` 2025-10-27 11:05:36.935 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:36.940 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW DATABASES 2025-10-27 11:05:36.949 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:36.955 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `db`, `user`, `select_priv`, `insert_priv`, `update_priv`, `delete_priv`, `create_priv`, `drop_priv`, `index_priv`, `alter_priv`, `create_view_priv`, `show_view_priv` , `host` FROM `mysql`.`db` 2025-10-27 11:05:36.993 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:36.999 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW DATABASES 2025-10-27 11:05:37.011 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:37.018 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ o.CREATED AS GMT_CREATE, o.OBJECT_ID AS DATABASE_ID, d.DATABASE_NAME, c.ID AS COLLATION_TYPE, NULL AS PRIMARY_ZONE, (CASE READ_ONLY WHEN 'YES' THEN 1 ELSE 0 END) AS READ_ONLY FROM oceanbase.DBA_OB_DATABASES d JOIN oceanbase.DBA_OBJECTS o JOIN information_schema.collations c ON BINARY d.DATABASE_NAME = BINARY o.OBJECT_NAME AND d.COLLATION = c.COLLATION_NAME WHERE o.OBJECT_TYPE = 'DATABASE' 2025-10-27 11:05:37.127 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.o.binlog.internal.CdcroUserService : create cdcro user skipped, user already exists, tenantId=5, username=cdcro 2025-10-27 11:05:37.132 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.o.b.i.t.CreateClusterCdcroUserTask : create cluster cdc user success, username=cdcro 2025-10-27 11:05:37.144 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9147, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:05:37.146 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,a3b9f55831e0ec9d] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9149] 2025-10-27 11:05:37.642 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b37b2b026096fdcc,58e64654bcfab812] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=b37b2b026096fdcc, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:05:37.642 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3bcfa9b9cc47eeb3,0860d0588fe6eac7] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=3bcfa9b9cc47eeb3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:05:37.642 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3bcfa9b9cc47eeb3,0860d0588fe6eac7] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=3bcfa9b9cc47eeb3, duration=7 ms] 2025-10-27 11:05:37.643 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,eab8766eded95ef3,359b6ecf1a90f18f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=eab8766eded95ef3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:05:37.647 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,007963399e858d83,a96cc982a055a0e3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.40.198.13, traceId=007963399e858d83, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-10-27 11:05:37.655 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,007963399e858d83,a96cc982a055a0e3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.40.198.13, traceId=007963399e858d83, duration=20 ms] 2025-10-27 11:05:37.655 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b37b2b026096fdcc,58e64654bcfab812] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=b37b2b026096fdcc, duration=21 ms] 2025-10-27 11:05:37.663 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:37.664 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:37.664 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:37.694 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,eab8766eded95ef3,359b6ecf1a90f18f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=eab8766eded95ef3, duration=59 ms] 2025-10-27 11:05:37.709 INFO 2458 --- [prometheus-cache-monitor1,4663ed4a13e81373,61c6db6a7a07678e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20297,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5075,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7860} 2025-10-27 11:05:37.710 INFO 2458 --- [prometheus-cache-monitor1,4663ed4a13e81373,61c6db6a7a07678e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:05:37.734 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,852d48719ca94d19] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9149, operation:EXECUTE, state: RUNNING 2025-10-27 11:05:37.762 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:05:37.770 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,992211856b2c9bc2,04851c7646cc4c0d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=992211856b2c9bc2, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:05:37.770 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9149, context=Context{parallelIdx=-1, stringMap={tenant_id=6, target_tenant_binlog_service_status=RUNNING, cluster_cdcro_password=xxx task_instance_id=9096, task_operation=execute, binlog_cluster_id=4, binlog_instance_replicate_count=1, tenant_binlog_service_id=9, operation_scope=Tenant, latest_execution_start_time=2025-10-27T11:05:37.732+08:00, sub_task_instance_name=Create tenant binlog service, sub_task_instance_id=9149, cluster_cdcro_username=cdcro}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, executor=10.40.65.146 2025-10-27 11:05:37.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,67b10c6600e70ad2,93eb501bd72e9f1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=67b10c6600e70ad2, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:05:37.845 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,67b10c6600e70ad2,93eb501bd72e9f1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=67b10c6600e70ad2, duration=32 ms] 2025-10-27 11:05:37.867 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:37.873 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:05:37.889 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:05:37.895 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: CREATE BINLOG FOR TENANT `myoceanbase2`.`test2` TO USER `cdcro` PASSWORD xxx WITH CLUSTER URL ?,REPLICATE NUM ?, args: [http://10.40.65.146:8081/services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2, 1] 2025-10-27 11:05:37.975 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,fb52a5d650d59720,56ee8ff49ede1816] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments?taskInstanceId=9096&subtaskInstanceId=9149, client=10.40.198.13, traceId=fb52a5d650d59720, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9096,9149,] 2025-10-27 11:05:37.975 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,992211856b2c9bc2,04851c7646cc4c0d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=992211856b2c9bc2, duration=212 ms] 2025-10-27 11:05:38.021 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,fb52a5d650d59720,56ee8ff49ede1816] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments client=10.40.198.13, traceId=fb52a5d650d59720, duration=53 ms] 2025-10-27 11:05:38.028 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,49d23e0af9650c4e,a63474664f6e676b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2, client=10.40.65.151, traceId=49d23e0af9650c4e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,false,0,] 2025-10-27 11:05:38.028 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,49d23e0af9650c4e,a63474664f6e676b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.151, traceId=49d23e0af9650c4e, duration=1 ms] 2025-10-27 11:05:38.078 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,d3a4a4b375cf171f,c64969f17bd512a0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2, client=10.40.65.151, traceId=d3a4a4b375cf171f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,false,0,] 2025-10-27 11:05:38.078 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,d3a4a4b375cf171f,c64969f17bd512a0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.151, traceId=d3a4a4b375cf171f, duration=0 ms] 2025-10-27 11:05:38.176 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] o.m.jdbc.message.server.ErrorPacket : Error: 1236-HY000: Node resources are insufficient 2025-10-27 11:05:38.267 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] update failed, sql: [CREATE BINLOG FOR TENANT `myoceanbase2`.`test2` TO USER `cdcro` PASSWORD xxx WITH CLUSTER URL ?,REPLICATE NUM ?], error message: [PreparedStatementCallback; uncategorized SQLException for SQL [CREATE BINLOG FOR TENANT `myoceanbase2`.`test2` TO USER `cdcro` PASSWORD xxx WITH CLUSTER URL ?,REPLICATE NUM ?]; SQL state [HY000]; error code [1236]; (conn=7898) Node resources are insufficient; nested exception is java.sql.SQLException: (conn=7898) Node resources are insufficient] 2025-10-27 11:05:38.273 ERROR 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : (conn=7898) Node resources are insufficient java.sql.SQLException: (conn=7898) Node resources are insufficient at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:306) at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:378) at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:187) at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:1364) at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:1303) at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:1222) at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:1146) at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:87) at org.mariadb.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:307) at org.mariadb.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:284) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1025) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.update(BinlogTemplate.java:80) at com.oceanbase.ocp.binlogsdk.operator.JdbcBinlogOperator.createTenantBinlogSpecifiedInstanceNum(JdbcBinlogOperator.java:96) at com.oceanbase.ocp.binlog.internal.BinlogOperationManager.createTenantBinlogService(BinlogOperationManager.java:238) at com.oceanbase.ocp.binlog.internal.task.CreateTenantBinlogServiceTask.run(CreateTenantBinlogServiceTask.java:58) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:212) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) 2025-10-27 11:05:38.283 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.c.t.e.c.w.subtask.SubtaskExecutor : Subtask execute failed. com.oceanbase.ocp.binlogsdk.operator.BinlogExecuteException: (conn=7898) Node resources are insufficient at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.customException(BinlogTemplate.java:121) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.customException(BinlogTemplate.java:113) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.update(BinlogTemplate.java:83) at com.oceanbase.ocp.binlogsdk.operator.JdbcBinlogOperator.createTenantBinlogSpecifiedInstanceNum(JdbcBinlogOperator.java:96) at com.oceanbase.ocp.binlog.internal.BinlogOperationManager.createTenantBinlogService(BinlogOperationManager.java:238) at com.oceanbase.ocp.binlog.internal.task.CreateTenantBinlogServiceTask.run(CreateTenantBinlogServiceTask.java:58) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.execute(JavaSubtaskRunner.java:64) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.doRun(JavaSubtaskRunner.java:32) at com.oceanbase.ocp.core.task.engine.runner.JavaSubtaskRunner.run(JavaSubtaskRunner.java:26) at com.oceanbase.ocp.core.task.engine.runner.RunnerFactory.doRun(RunnerFactory.java:76) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.doRun(SubtaskExecutor.java:212) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.redirectConsoleOutput(SubtaskExecutor.java:206) at com.oceanbase.ocp.core.task.engine.coordinator.worker.subtask.SubtaskExecutor.lambda$submit$2(SubtaskExecutor.java:137) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.sql.SQLException: (conn=7898) Node resources are insufficient at org.mariadb.jdbc.export.ExceptionFactory.createException(ExceptionFactory.java:306) at org.mariadb.jdbc.export.ExceptionFactory.create(ExceptionFactory.java:378) at org.mariadb.jdbc.message.ClientMessage.readPacket(ClientMessage.java:187) at org.mariadb.jdbc.client.impl.StandardClient.readPacket(StandardClient.java:1364) at org.mariadb.jdbc.client.impl.StandardClient.readResults(StandardClient.java:1303) at org.mariadb.jdbc.client.impl.StandardClient.readResponse(StandardClient.java:1222) at org.mariadb.jdbc.client.impl.StandardClient.execute(StandardClient.java:1146) at org.mariadb.jdbc.ClientPreparedStatement.executeInternal(ClientPreparedStatement.java:87) at org.mariadb.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:307) at org.mariadb.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:284) at org.springframework.jdbc.core.JdbcTemplate.lambda$update$2(JdbcTemplate.java:965) at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:648) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:960) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1015) at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:1025) at com.oceanbase.ocp.binlogsdk.operator.BinlogTemplate.update(BinlogTemplate.java:80) ... 14 common frames omitted 2025-10-27 11:05:38.291 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,99290cbc9df6f3c3] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9149, operation:EXECUTE, state: FAILED 2025-10-27 11:05:38.476 INFO 2458 --- [ocp-async-2,bc020e85ecabee0f,e88dd3e87eb22a74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:38.476 WARN 2458 --- [ocp-async-3,eab171ec81049de4,9a0ecdb11615d9d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:38.507 INFO 2458 --- [ocp-async-5,1f970dade86152f7,8bd44739634b3107] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:38.571 INFO 2458 --- [ocp-schedules-9,0e5de399c8a83d2e,277b3e289fa4d451] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:05:38.577 INFO 2458 --- [ocp-schedules-9,0e5de399c8a83d2e,277b3e289fa4d451] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:38.577 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:05:38.577 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:05:38.578 INFO 2458 --- [ocp-schedules-9,0e5de399c8a83d2e,277b3e289fa4d451] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:05:38.656 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0ea9de5b3671a3c2,9494ae924c3bc6ac] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=0ea9de5b3671a3c2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:05:38.656 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0ea9de5b3671a3c2,9494ae924c3bc6ac] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=0ea9de5b3671a3c2, duration=0 ms] 2025-10-27 11:05:38.699 INFO 2458 --- [ocp-schedules-21,f8f1bcb304c4f4aa,f6e02def9f7e54ae] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:38.707 INFO 2458 --- [prometheus-minute-schedule-1,3c7d5291725bd762,1ae4901bbb311336] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:38.719 INFO 2458 --- [prometheus-minute-schedule-1,3c7d5291725bd762,1ae4901bbb311336] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:38.736 WARN 2458 --- [scanner-112129061-1,f8f1bcb304c4f4aa,c76195c97aa96554] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:38.916 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,23900c18bdecc53d,c184efb1875433cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=23900c18bdecc53d, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:05:38.938 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,23900c18bdecc53d,c184efb1875433cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=23900c18bdecc53d, duration=32 ms] 2025-10-27 11:05:38.946 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:38.947 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:38.995 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b2d7e020194cccf6,d4770fba7cd24b5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments?taskInstanceId=9096&subtaskInstanceId=9149, client=10.40.198.13, traceId=b2d7e020194cccf6, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9096,9149,] 2025-10-27 11:05:39.000 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b2d7e020194cccf6,d4770fba7cd24b5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments client=10.40.198.13, traceId=b2d7e020194cccf6, duration=11 ms] 2025-10-27 11:05:39.363 INFO 2458 --- [task-coordinator2,,] c.o.o.c.t.e.c.worker.RunningTaskWorker : Mark failed task, entity=TaskInstanceEntity{id=9096, name='Create tenant binlog service', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=6, hostId=null, creator='admin', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={binlog_instance_replicate_count=1, tenant_id=6, tenant_binlog_service_id=9, operation_scope=Tenant, prohibit_rollback=false, binlog_cluster_id=4}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, createTime=2025-10-27T11:05:34+08:00, startTime=2025-10-27T11:05:34.708+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=9150, name=Update tenant binlog service status, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.UpdateTenantBinlogServiceStatusWhenCreateTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=9149, name=Create tenant binlog service, state=FAILED, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.CreateTenantBinlogServiceTask, seriesId=7, startTime=2025-10-27T11:05:37.733+08:00, endTime=2025-10-27T11:05:38.284+08:00}, SubtaskInstanceEntity{id=9148, name=Update cdcro user info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.UpdateClusterCdcroUserInfoTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=9146, name=Prepare create tenant binlog service, state=SUCCESSFUL, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.PrepareCreateTenantBinlogServiceTask, seriesId=5, startTime=2025-10-27T11:05:35.696+08:00, endTime=2025-10-27T11:05:35.751+08:00}, SubtaskInstanceEntity{id=9152, name=Update tenant binlog service info, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.UpdateBinlogInstanceInfoWhenActiveBinlogServiceTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=9151, name=Set binlog instance parameter, state=PENDING, operation=EXECUT 2025-10-27 11:05:39.417 WARN 2458 --- [task-coordinator2,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm type ob_tenant_operation_info is not enabled 2025-10-27 11:05:39.418 INFO 2458 --- [task-coordinator2,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"status":"Disabled"} 2025-10-27 11:05:39.418 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9096, status:FAILED 2025-10-27 11:05:39.627 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,fe70274c7bd883bb,0750848ed2b4f101] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=fe70274c7bd883bb, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:05:39.627 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,fe70274c7bd883bb,0750848ed2b4f101] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=fe70274c7bd883bb, duration=9 ms] 2025-10-27 11:05:39.630 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,c0116585259c2b18,e954453dfd4e83a4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=c0116585259c2b18, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:05:39.630 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,c0116585259c2b18,e954453dfd4e83a4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=c0116585259c2b18, duration=12 ms] 2025-10-27 11:05:39.631 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,df7c23896d21548e,4baf45259c230a8f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=df7c23896d21548e, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:05:39.631 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,df7c23896d21548e,4baf45259c230a8f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=df7c23896d21548e, duration=13 ms] 2025-10-27 11:05:39.652 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:05:39.653 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:05:39.653 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:05:40.010 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:05:40.010 INFO 2458 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:05:40.012 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,abbf6e81d7c92fa9,9d6230ffaf677e15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=abbf6e81d7c92fa9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:05:40.036 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,abbf6e81d7c92fa9,9d6230ffaf677e15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=abbf6e81d7c92fa9, duration=35 ms] 2025-10-27 11:05:40.050 INFO 2458 --- [Collect obproxy1,cd8525f0afb9f8e8,d81ec64fcbbb1c15] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:05:40.050 INFO 2458 --- [Check obproxy1,c5ec01bcefa29b88,76ff310f3bdf55d3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:05:40.054 INFO 2458 --- [Collect obproxy1,cd8525f0afb9f8e8,d81ec64fcbbb1c15] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:05:40.055 INFO 2458 --- [Check obproxy1,c5ec01bcefa29b88,76ff310f3bdf55d3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:05:40.084 INFO 2458 --- [Collect obproxy1,cd8525f0afb9f8e8,d81ec64fcbbb1c15] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:05:40.084 INFO 2458 --- [Check obproxy1,c5ec01bcefa29b88,76ff310f3bdf55d3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:05:40.087 INFO 2458 --- [Check obproxy1,c5ec01bcefa29b88,76ff310f3bdf55d3] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:05:40.087 INFO 2458 --- [Check obproxy1,c5ec01bcefa29b88,76ff310f3bdf55d3] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:05:40.088 INFO 2458 --- [Collect obproxy1,cd8525f0afb9f8e8,d81ec64fcbbb1c15] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:05:40.090 INFO 2458 --- [Collect obproxy1,cd8525f0afb9f8e8,d81ec64fcbbb1c15] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:05:40.214 INFO 2458 --- [ocp-schedules-20,b2ab9e7ed094921b,cbc8a1f78e48250d] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:05:40.215 INFO 2458 --- [ocp-schedules-20,b2ab9e7ed094921b,cbc8a1f78e48250d] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:05:40.239 INFO 2458 --- [ocp-schedules-20,b2ab9e7ed094921b,cbc8a1f78e48250d] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:05:40.476 WARN 2458 --- [ocp-async-5,824aeb364fc0f18c,89bea752a77ecea8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:40.476 INFO 2458 --- [ocp-async-8,9e09378090ecb60b,5fd5b17289f73720] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:41.707 INFO 2458 --- [prometheus-minute-schedule-1,dc55a51f31f7f96b,a53b96680ff791c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:41.713 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:41.713121+08:00, lastProcessTime=2025-10-27T11:05:31.714067+08:00 2025-10-27 11:05:41.714 INFO 2458 --- [prometheus-minute-schedule-1,dc55a51f31f7f96b,a53b96680ff791c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:41.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:41.713121+08:00 2025-10-27 11:05:41.947 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:41.948 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:42.476 INFO 2458 --- [ocp-async-8,0c1b09f595c41c6a,ff027dd9679c9501] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:42.476 WARN 2458 --- [ocp-async-4,4fe7e14242d54ee5,5a41b313a0858998] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:42.622 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5bdbe2252ded599e,16f31bf213994794] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=5bdbe2252ded599e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:05:42.622 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,5bdbe2252ded599e,16f31bf213994794] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=5bdbe2252ded599e, duration=9 ms] 2025-10-27 11:05:42.623 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,58b7c21cc1ee4d95,4b6e30eb9145df41] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=58b7c21cc1ee4d95, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:05:42.627 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,58b7c21cc1ee4d95,4b6e30eb9145df41] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=58b7c21cc1ee4d95, duration=14 ms] 2025-10-27 11:05:43.436 INFO 2458 --- [ocp-schedules-19,bea579d13f92b85b,1e77e48c7c1cc880] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:43.440 INFO 2458 --- [ocp-schedules-19,bea579d13f92b85b,1e77e48c7c1cc880] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:43.963 INFO 2458 --- [ocp-schedules-17,af9524ee721b4171,34cb62f24988f4c3] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:05:44.170 INFO 2458 --- [ocp-schedules-4,7931967b2c7a6421,5642b3be3b9329e7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:05:44.170 INFO 2458 --- [ocp-schedules-4,7931967b2c7a6421,5642b3be3b9329e7] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:05:44.177 INFO 2458 --- [ocp-schedules-4,7931967b2c7a6421,5642b3be3b9329e7] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:05:44.177 INFO 2458 --- [ocp-schedules-4,7931967b2c7a6421,5642b3be3b9329e7] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:05:44.476 INFO 2458 --- [ocp-async-8,a27c59fd4259a5b6,db5188c730256e94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:44.476 WARN 2458 --- [ocp-async-1,eb5ee8b54ac675bd,6dd44fc4fea29bcf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:44.707 INFO 2458 --- [prometheus-minute-schedule-1,1fdb977fe7810f3f,5c8338126d952696] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:44.716 INFO 2458 --- [prometheus-minute-schedule-1,1fdb977fe7810f3f,5c8338126d952696] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:44.948 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:44.949 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:45.102 INFO 2458 --- [prometheus-second-schedule-1,37f919ee99065ce1,4f3874aac8257da8] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:05:45.750 INFO 2458 --- [ocp-schedules-25,e506bf152e7c180a,2dccdae852ff4948] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:46.476 WARN 2458 --- [ocp-async-5,1432e9e9ff8a9d37,bf36cdb18f01a39c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:46.476 INFO 2458 --- [ocp-async-8,7e5bc65a52736a7c,18b277c6aff5ef35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:46.624 INFO 2458 --- [ocp-schedules-7,da34c5bc37532cd2,de5525a8a4dd5905] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:05:46.632 INFO 2458 --- [ocp-schedules-7,da34c5bc37532cd2,de5525a8a4dd5905] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:46.632 INFO 2458 --- [ocp-schedules-7,da34c5bc37532cd2,de5525a8a4dd5905] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:05:47.544 INFO 2458 --- [ocp-schedules-2,a1c3881c4487dc9c,8290056d1df3983a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:05:47.544+08:00, clusters=[1, 2] 2025-10-27 11:05:47.583 INFO 2458 --- [ocp-schedules-2,a1c3881c4487dc9c,8290056d1df3983a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:05:47.585 INFO 2458 --- [ocp-schedules-2,a1c3881c4487dc9c,8290056d1df3983a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:05:47.707 INFO 2458 --- [prometheus-minute-schedule-1,63a3306dabcc8701,7e76bcedbad8809e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:47.715 INFO 2458 --- [prometheus-minute-schedule-1,63a3306dabcc8701,7e76bcedbad8809e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:47.949 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:47.949 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:48.477 INFO 2458 --- [ocp-async-8,ae17befcb544a69c,13a00503626bdb41] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:48.477 WARN 2458 --- [ocp-async-4,0c4a790244e21604,9586dc6e497e401a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:48.508 INFO 2458 --- [ocp-async-1,21662ccd85d7fc64,488e327c9ab25d70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:48.826 INFO 2458 --- [ocp-schedules-30,1e4c5c9f3fbf9b34,87dc8fc0542e7926] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:48.872 WARN 2458 --- [scanner-112129280-1,1e4c5c9f3fbf9b34,b4103dad0b3763b1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:48.949 INFO 2458 --- [alarm-detect-19,1e4c5c9f3fbf9b34,3e58d2142ffad8ab] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534278 ~ 1761534338 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=202634.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202634.56, values={1761534300=202634.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:05:48.949 INFO 2458 --- [alarm-detect-19,1e4c5c9f3fbf9b34,3e58d2142ffad8ab] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:05:48.983 INFO 2458 --- [alarm-detect-13,1e4c5c9f3fbf9b34,c7e70396ec453d29] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534278 ~ 1761534338 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=92.951)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.951, values={1761534300=92.951}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:05:48.984 INFO 2458 --- [alarm-detect-13,1e4c5c9f3fbf9b34,c7e70396ec453d29] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:05:49.011 INFO 2458 --- [alarm-detect-7,1e4c5c9f3fbf9b34,35af5c66cc680017] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534278 ~ 1761534338 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=202635.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202635.912, values={1761534300=202635.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:05:49.012 INFO 2458 --- [alarm-detect-7,1e4c5c9f3fbf9b34,35af5c66cc680017] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:05:49.032 INFO 2458 --- [alarm-detect-6,1e4c5c9f3fbf9b34,6ca75df7f0459e43] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534315 ~ 1761534339 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534330, value=51.168)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534330, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534330=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:05:50.477 WARN 2458 --- [ocp-async-1,78c8f8009dcf6ca9,7cb48d8a01e21e81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:50.477 INFO 2458 --- [ocp-async-6,16d6dcf9b1c8df8b,a151caa5a76a68f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:50.707 INFO 2458 --- [prometheus-minute-schedule-1,cb42343e373c619d,24c6eb0b44f146cd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:50.715 INFO 2458 --- [prometheus-minute-schedule-1,cb42343e373c619d,24c6eb0b44f146cd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:50.950 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:50.950 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:51.409 INFO 2458 --- [ocp-schedules-7,8a100e113f6017ef,3a46ffbb18a23d90] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:51.409 INFO 2458 --- [ocp-schedules-20,a8b610da1dc1f9eb,af047474137568b4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:51.409 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:05:51.409 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:05:51.409 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:05:51.409 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:05:51.409 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:05:51.410 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:05:51.410 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:05:51.410 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:05:51.410 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:05:51.410 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:05:51.424 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:51.425 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:51.426 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:51.426 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:51.427 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:51.427 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:51.427 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:51.427 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:51.429 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:51.430 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:51.431 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:51.431 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:51.431 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:51.432 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:51.432 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:51.432 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:51.433 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:51.434 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:51.436 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:51.436 INFO 2458 --- [ocp-schedules-28,22e3be32886a3088,8f21991154797ee0] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:05:51.436 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:51.436 INFO 2458 --- [host-check-clockdiff1,22e3be32886a3088,393387670765d37d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:05:51.436 INFO 2458 --- [host-check-clockdiff2,22e3be32886a3088,05e62c6212c80da1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:05:51.437 INFO 2458 --- [host-check-clockdiff3,22e3be32886a3088,3a24b51aa70bb457] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:05:51.437 INFO 2458 --- [host-check-clockdiff4,22e3be32886a3088,4842925fa8bd8aa9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:05:51.437 INFO 2458 --- [host-check-clockdiff5,22e3be32886a3088,9bd8088a6cb5bd7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:05:51.442 INFO 2458 --- [ocp-schedules-1,de108f7674b66e39,7212ed2bca57fde6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:05:51.461 INFO 2458 --- [host-check-clockdiff2,22e3be32886a3088,05e62c6212c80da1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534351 -1 -1 , error= 2025-10-27 11:05:51.462 INFO 2458 --- [host-check-clockdiff1,22e3be32886a3088,393387670765d37d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534351 0 0 , error= 2025-10-27 11:05:51.464 INFO 2458 --- [host-check-clockdiff3,22e3be32886a3088,3a24b51aa70bb457] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534351 -1 -1 , error= 2025-10-27 11:05:51.503 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:05:51.503 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:05:51.504 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:05:51.507 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:05:51.507 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:05:51.507372379+08:00 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-agent-status3,a8b610da1dc1f9eb,be49eace6f8dae37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:05:51.509 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:05:51.509+08:00 2025-10-27 11:05:51.510 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:05:51.509411070+08:00 2025-10-27 11:05:51.510 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:05:51.510 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:05:51.510107156+08:00 2025-10-27 11:05:51.511 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:05:51.510+08:00 2025-10-27 11:05:51.511 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:05:51.510+08:00 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status2,a8b610da1dc1f9eb,11b286151960a235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:05:51.513 INFO 2458 --- [host-check-agent-status5,a8b610da1dc1f9eb,0f95589d4c93f23a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:05:51.514 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:05:51.514 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:05:51.514 INFO 2458 --- [host-check-agent-status4,a8b610da1dc1f9eb,233e0a2735ea4690] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:05:51.521 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:05:51.521 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:05:51.523 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:05:51.521743012+08:00 2025-10-27 11:05:51.524 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:05:51.524+08:00 2025-10-27 11:05:51.526 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:05:51.527 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:05:51.509+08:00 and status AVAILABLE 2025-10-27 11:05:51.529 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:05:51.528194455+08:00 2025-10-27 11:05:51.530 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:05:51.529+08:00 2025-10-27 11:05:51.530 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:05:51.510+08:00 and status AVAILABLE 2025-10-27 11:05:51.532 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:05:51.510+08:00 and status AVAILABLE 2025-10-27 11:05:51.534 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:05:51.524+08:00 and status AVAILABLE 2025-10-27 11:05:51.534 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:05:51.536 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:05:51.537 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:05:51.537 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:05:51.537 INFO 2458 --- [host-check-agent-status1,a8b610da1dc1f9eb,f444620aa465d534] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:05:51.537 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:05:51.540 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:05:51.529+08:00 and status AVAILABLE 2025-10-27 11:05:51.540 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:05:51.544 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:05:51.545 INFO 2458 --- [ocp-schedules-24,a75a113df026c532,efc9fa9e327e73d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:05:51.549 INFO 2458 --- [ocp-schedules-23,2ba308484e6e237c,c9a3d9d26a096dec] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:05:51.551 INFO 2458 --- [ocp-schedules-24,a75a113df026c532,efc9fa9e327e73d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:51.553 INFO 2458 --- [ocp-schedules-23,2ba308484e6e237c,c9a3d9d26a096dec] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:51.554 INFO 2458 --- [ocp-schedules-23,2ba308484e6e237c,c9a3d9d26a096dec] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:05:51.554 INFO 2458 --- [ocp-schedules-23,c2ff286c3ed07c8b,8dd8df74ec866197] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:05:51.558 INFO 2458 --- [ocp-schedules-23,c2ff286c3ed07c8b,8dd8df74ec866197] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:51.565 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:05:51.565 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:05:51.566 INFO 2458 --- [ocp-schedules-9,c3b43d9e8c9294b1,1afe5a3d583044d8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:05:51.566 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:51.567 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:51.568 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:05:51.568 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:05:51.569 INFO 2458 --- [ocp-schedules-30,3ba4bac5af11de1d,685890ba79c975e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:05:51.569 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:51.569 INFO 2458 --- [ocp-schedules-17,1f87576ae9dd5e15,e0abb6ce111421db] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:05:51.570 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:51.571 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:05:51.574 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:05:51.578 INFO 2458 --- [ocp-schedules-30,3ba4bac5af11de1d,685890ba79c975e6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:05:51.579 INFO 2458 --- [ocp-schedules-17,1f87576ae9dd5e15,e0abb6ce111421db] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:05:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:51.602 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:05:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.632 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:51.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:51.643 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:51.645 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:51.646 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:51.648 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:51.652 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:05:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:05:51.654 INFO 2458 --- [ocp-schedules-24,a75a113df026c532,efc9fa9e327e73d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:05:51.664 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:05:51.666 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:05:51.667 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:05:51.668 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:05:51.678 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:05:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.697 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:51.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.698 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:05:51.698 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:05:51.699 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:05:51.702 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:05:51.706 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:05:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:05:51.715140+08:00, lastProcessTime=2025-10-27T11:05:41.713121+08:00 2025-10-27 11:05:51.717 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:05:51.749 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:51.749 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:51.750 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:51.750 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:51.757 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:05:51.765 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:05:51.766 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:51.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:51.769 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:05:51.770 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:05:51.770 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:05:51.772 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:05:51.777 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:51.788 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:05:51.795 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:05:51.798 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:05:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.800 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:05:51.801 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.801 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:05:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:51.807 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:05:51.818 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:05:51.838 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:51.839 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:51.840 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:51.844 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:51.845 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:51.852 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:05:51.854 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:05:51.855 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:05:51.865 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:05:51.869 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:05:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:51.871 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.872 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:51.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:05:51.890 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:05:51.891 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:05:51.898 INFO 2458 --- [host-check-host-status2,8a100e113f6017ef,c16b6990b204d9f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:05:51.901 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:05:51.902 INFO 2458 --- [host-check-host-status4,8a100e113f6017ef,60823bc1f8e36a69] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:05:51.903 INFO 2458 --- [host-check-host-status3,8a100e113f6017ef,3b97dbe4f2a2c50f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:05:51.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:05:51.909 INFO 2458 --- [host-check-host-status5,8a100e113f6017ef,0e8584fb2ef36666] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:05:51.930 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:51.930 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.930 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:05:51.934 INFO 2458 --- [host-check-host-status1,8a100e113f6017ef,70da190b44e7a3bf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:05:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:51.935 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:51.938 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:05:51.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:51.968 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:05:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:52.010 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:05:52.011 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.012 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:05:52.017 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.018 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:05:52.021 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:05:51.715140+08:00 2025-10-27 11:05:52.023 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:52.023 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:52.039 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:05:52.058 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:52.058 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:52.065 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:52.069 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.069 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:05:52.092 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:05:52.092 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:05:52.097 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:52.097 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:05:52.133 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:52.133 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:05:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:05:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.217 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.217 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:05:52.221 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:05:52.231 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:05:52.240 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:05:52.243 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:05:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:05:52.340 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:05:52.340 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.340 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:05:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:05:52.382 INFO 2458 --- [host-check-clockdiff4,22e3be32886a3088,4842925fa8bd8aa9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534352 0 0 , error= 2025-10-27 11:05:52.390 INFO 2458 --- [host-check-clockdiff5,22e3be32886a3088,9bd8088a6cb5bd7c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534352 0 0 , error= 2025-10-27 11:05:52.390 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:52.432 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:52.433 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:05:52.475 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:05:52.475 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:05:52.475 INFO 2458 --- [ocp-schedules-23,c2ff286c3ed07c8b,8dd8df74ec866197] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:05:52.477 WARN 2458 --- [ocp-async-10,2be569b080dacfb1,161293e25cc3ec03] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:52.477 INFO 2458 --- [ocp-async-7,7458f70de824f90e,f7cd755c050bcbdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:05:53.219 INFO 2458 --- [ocp-schedules-30,dfb7433156d96501,1c338f7eda9c7bb4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20297,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5075,"seriesIdKeySize":8769,"secondWriteQueue":0,"minuteWriteCacheSize":7860} 2025-10-27 11:05:53.220 INFO 2458 --- [ocp-schedules-30,dfb7433156d96501,1c338f7eda9c7bb4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:05:53.447 INFO 2458 --- [ocp-schedules-19,b8010f5badc0a5af,b3dd403b8d9aa9ad] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:05:53.451 INFO 2458 --- [ocp-schedules-19,b8010f5badc0a5af,b3dd403b8d9aa9ad] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:05:53.707 INFO 2458 --- [prometheus-minute-schedule-1,60e28c0976408ffe,00f719e6d081a0bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:53.715 INFO 2458 --- [prometheus-minute-schedule-1,60e28c0976408ffe,00f719e6d081a0bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:53.950 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:53.951 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:54.330 INFO 2458 --- [ocp-schedules-9,d50ed0d8415278a8,d9450fc33c09e66e] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:05:54.477 WARN 2458 --- [ocp-async-3,cb9297e014390f00,27cf6307b92a7971] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:54.477 INFO 2458 --- [ocp-async-3,11155a6daa61b96d,0c8ec5dac97ab071] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:55.616 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,e40cb0b3c815db81,9b520bf637de42f3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=e40cb0b3c815db81, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:05:55.616 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,e40cb0b3c815db81,9b520bf637de42f3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=e40cb0b3c815db81, duration=1 ms] 2025-10-27 11:05:55.759 INFO 2458 --- [ocp-schedules-1,b5e6899b7f4a5407,fadafe652ee5c33c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:05:56.450 INFO 2458 --- [ocp-schedules-27,f81fc0a6bbe70a10,d4878be42bc876a9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:05:56.458 INFO 2458 --- [ocp-schedules-27,f81fc0a6bbe70a10,d4878be42bc876a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:56.477 WARN 2458 --- [ocp-async-6,50663bb901624125,692f4b754a998d79] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:56.477 INFO 2458 --- [ocp-async-5,d08c0685b3a89b56,4af3e41fccccbf6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:56.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:05:56.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.546 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:05:56.557 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.557 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.582 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.583 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.584 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.586 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:05:56.602 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.602 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.611 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.612 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.614 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.615 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.643 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.644 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.659 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.659 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.659 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.660 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:05:56.662 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.662 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.663 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.663 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.670 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.671 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.691 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.691 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.692 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.692 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.702 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.703 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.704 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.704 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.705 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.706 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.707 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.707 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.707 INFO 2458 --- [prometheus-minute-schedule-1,6d501d8216842253,47ffb28d87631efd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:56.708 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.709 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.709 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.710 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.711 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:05:56.713 INFO 2458 --- [prometheus-minute-schedule-1,6d501d8216842253,47ffb28d87631efd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:56.718 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.718 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.719 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:05:56.723 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:05:56.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.724 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.724 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.725 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:05:56.754 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.754 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.754 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.755 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:05:56.770 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:56.771 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.771 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:05:56.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:05:56.783 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:05:56.787 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.787 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.787 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.788 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:05:56.788 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:05:56.788 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:05:56.813 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.813 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.813 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.814 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:05:56.840 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.840 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.840 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.840 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:05:56.885 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.885 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.886 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.887 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.887 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.887 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.887 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.888 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.889 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.889 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.892 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.893 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.894 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.894 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.896 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.897 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.897 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.897 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.897 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.898 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.899 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.899 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.899 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.900 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.900 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.900 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.900 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.901 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.902 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.902 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.902 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.903 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.903 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.903 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.903 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.905 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.905 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.906 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:05:56.907 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.907 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:05:56.908 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.908 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.908 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.909 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:05:56.909 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:05:56.912 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:05:56.951 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:56.952 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:05:56.992 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:05:56.993 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.993 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.993 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:05:56.993 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:05:56.993 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:05:56.994 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:05:57.020 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:05:57.026 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:05:57.026 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:05:57.026 INFO 2458 --- [ocp-schedules-27,f81fc0a6bbe70a10,d4878be42bc876a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:05:57.641 INFO 2458 --- [ocp-schedules-5,8ff6f733dd22ca6f,9989811354545561] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:05:57.648 INFO 2458 --- [ocp-schedules-5,8ff6f733dd22ca6f,9989811354545561] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:05:57.690 INFO 2458 --- [ocp-schedules-5,8ff6f733dd22ca6f,9989811354545561] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:05:58.477 INFO 2458 --- [ocp-async-4,966030d64e1e84b8,d56ee1f9ac4d18fd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:05:58.477 WARN 2458 --- [ocp-async-10,a191a7887b598d93,c45ba2e3e46addc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:05:58.508 INFO 2458 --- [ocp-async-7,32fe10cf88752d0a,d25bbc62dc39ab4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:05:59.046 INFO 2458 --- [ocp-schedules-29,036a773b10fe9f25,1bdae59ff9bcb069] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:05:59.099 WARN 2458 --- [scanner-112129532-1,036a773b10fe9f25,857c4223451748bf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:05:59.160 INFO 2458 --- [ocp-schedules-28,545138734c4ae533,8a1d385e37bf97fb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:05:59.200 INFO 2458 --- [alarm-detect-7,036a773b10fe9f25,6b863b298cd73f91] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534330 ~ 1761534349 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534345, value=44.161)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534345, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534345=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:05:59.707 INFO 2458 --- [prometheus-minute-schedule-1,5fa9fb96a28e8be3,8669855ec7f36301] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534360, interval=60 2025-10-27 11:05:59.715 INFO 2458 --- [prometheus-minute-schedule-1,5fa9fb96a28e8be3,8669855ec7f36301] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:05:59.952 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:05:59.953 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:00.477 WARN 2458 --- [ocp-async-7,4d119c2250aee0b0,ad79f1019e0dff93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:00.477 INFO 2458 --- [ocp-async-8,bcc8c936779ca152,0f9813430a977ad3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:01.713251+08:00, lastProcessTime=2025-10-27T11:05:51.715140+08:00 2025-10-27 11:06:01.714 INFO 2458 --- [ocp-schedules-25,88d9b343dfb220cd,f6516cc6bb4bff38] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,,C_binlogtest2-4,C_binlogtest2-4/H_10.40.65.151,C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY,C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188/H_10.40.66.107,C_obproxyTest-1,C_obproxyTest-1/H_10.40.67.22,C_obproxyTest-1/H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22, current targets ,,C_binlogtest2-4,C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2,C_binlogtest2-4/H_10.40.65.151,C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY,C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188/H_10.40.66.107,C_obproxyTest-1,C_obproxyTest-1/H_10.40.67.22,C_obproxyTest-1/H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:01.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:01.713251+08:00 2025-10-27 11:06:01.959 INFO 2458 --- [ocp-schedules-18,6216cd1afd5364f3,f532b291504276f1] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:06:01.964 INFO 2458 --- [ocp-schedules-18,6216cd1afd5364f3,f532b291504276f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:02.000 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:02.001 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:06:02.016 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:02.016 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:02.016 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:02.016 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:02.016 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:02.017 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:06:02.032 INFO 2458 --- [ocp-schedules-18,6216cd1afd5364f3,f532b291504276f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:06:02.478 INFO 2458 --- [ocp-async-6,5469f583cae50034,0fa2c3924f48c357] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:02.478 WARN 2458 --- [ocp-async-5,21b696dd2ee9af8b,8f7ac58e7d0095ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:02.707 INFO 2458 --- [prometheus-minute-schedule-1,7f3b0e0ed0a9ee3f,b011b98eaace773e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:02.715 INFO 2458 --- [prometheus-minute-schedule-1,7f3b0e0ed0a9ee3f,b011b98eaace773e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:02.953 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:02.954 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:03.459 INFO 2458 --- [ocp-schedules-18,126f440461ffcd9a,9635efb034f34813] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:03.462 INFO 2458 --- [ocp-schedules-18,126f440461ffcd9a,9635efb034f34813] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:04.478 INFO 2458 --- [ocp-async-9,6be864f5095815ec,a900ba28f3404498] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:04.478 WARN 2458 --- [ocp-async-1,076dc68d9db77eda,d5676bd32e48c154] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:05.707 INFO 2458 --- [prometheus-minute-schedule-1,1647dfa4d3dc849b,cd58d13b2e420af5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:05.716 INFO 2458 --- [prometheus-minute-schedule-1,1647dfa4d3dc849b,cd58d13b2e420af5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:05.767 INFO 2458 --- [ocp-schedules-2,199f81f8e9ec5014,714ace8d032875c8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:05.954 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:05.955 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:06.023 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:06.024 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:06:06.025 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:06:06.027 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:06:06.088 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:06:06.131 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:06:06.154 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:06:06.154 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:06:06.154 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:06:06.154 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:06:06.157 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:06:06.158 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:06.159 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:06:06.160 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:06.161 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:06:06.383 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:03:30, start=2025-10-27T11:00:00, end=2025-10-27T11:02:30, updates=208, granule=120000ms, delay=210000ms 2025-10-27 11:06:06.394 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:03:30, target=2025-10-27T11:02:30, granule=120000ms, delay=60000ms) 2025-10-27 11:06:06.394 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:02:30, target=2025-10-27T10:52:30, granule=600000ms, delay=600000ms) 2025-10-27 11:06:06.394 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=367ms, period=120000ms 2025-10-27 11:06:06.478 INFO 2458 --- [ocp-async-7,30a8accc9a70d9c3,eec7980e30ed6f71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:06.478 WARN 2458 --- [ocp-async-3,dfde0d88f65ef143,ee68b0b9bb22a87e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:06.634 INFO 2458 --- [ocp-schedules-16,d8daf775a687b2a3,62680916917b07c1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:06:06.642 INFO 2458 --- [ocp-schedules-16,d8daf775a687b2a3,62680916917b07c1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:06.642 INFO 2458 --- [ocp-schedules-16,d8daf775a687b2a3,62680916917b07c1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:07.584 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:06:07.584+08:00, tenantSize=6 2025-10-27 11:06:07.613 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761534367584000 2025-10-27 11:06:07.613 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761534067405000 2025-10-27 11:06:07.694 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:06:07.704 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761534367698000 2025-10-27 11:06:07.704 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761534067539000 2025-10-27 11:06:07.728 INFO 2458 --- [ocp-schedules-10,848679bbc56c6b70,ec3a14c04968063c] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:06:08.478 INFO 2458 --- [ocp-async-5,6437335245133e71,0e63a9c3f7fcb415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:08.478 WARN 2458 --- [ocp-async-6,485e2176684fa237,9fd4853dc6b8039c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:08.508 INFO 2458 --- [ocp-async-10,ed2d65e06e1e0c5d,84bd6f52dc1bebb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:08.707 INFO 2458 --- [prometheus-minute-schedule-1,98d9e230ba4fcd20,25061a8ef057110a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:08.718 INFO 2458 --- [prometheus-minute-schedule-1,98d9e230ba4fcd20,25061a8ef057110a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:08.955 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:08.955 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:09.205 INFO 2458 --- [ocp-schedules-21,3f104d984935e75e,7f22a88fb28c8df8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:06:09.244 WARN 2458 --- [scanner-112129704-1,3f104d984935e75e,a5771208dbfb579b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:06:10.014 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:06:10.015 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:06:10.068 INFO 2458 --- [Check obproxy1,da0e0567d702758f,c87fd2edaa756891] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:06:10.068 INFO 2458 --- [Collect obproxy1,87b9dc3e1d14759c,be3c43b056f996d5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:06:10.068 INFO 2458 --- [Check obproxy1,da0e0567d702758f,c87fd2edaa756891] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:06:10.068 INFO 2458 --- [Collect obproxy1,87b9dc3e1d14759c,be3c43b056f996d5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:06:10.098 INFO 2458 --- [Check obproxy1,da0e0567d702758f,c87fd2edaa756891] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:06:10.100 INFO 2458 --- [Collect obproxy1,87b9dc3e1d14759c,be3c43b056f996d5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:06:10.101 INFO 2458 --- [Check obproxy1,da0e0567d702758f,c87fd2edaa756891] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:06:10.101 INFO 2458 --- [Check obproxy1,da0e0567d702758f,c87fd2edaa756891] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:06:10.104 INFO 2458 --- [Collect obproxy1,87b9dc3e1d14759c,be3c43b056f996d5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:06:10.106 INFO 2458 --- [Collect obproxy1,87b9dc3e1d14759c,be3c43b056f996d5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:06:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:06:10.478 INFO 2458 --- [ocp-async-1,470eda718adf09f8,3ca04d4f0bf6e469] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:10.478 WARN 2458 --- [ocp-async-9,b184547b59d505a3,5fd31c6d4c559737] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:11.707 INFO 2458 --- [prometheus-minute-schedule-1,9921dc18e755e6e1,0613653952171132] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:11.717 INFO 2458 --- [prometheus-minute-schedule-1,9921dc18e755e6e1,0613653952171132] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:11.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:11.717466+08:00, lastProcessTime=2025-10-27T11:06:01.713251+08:00 2025-10-27 11:06:11.956 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:11.956 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:12.266 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:11.717466+08:00 2025-10-27 11:06:12.478 INFO 2458 --- [ocp-async-3,2dbd021ae8d06d7d,3ca6bab2c0640591] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:12.478 WARN 2458 --- [ocp-async-7,69133b5d061c53cf,f695641b562c63f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:13.064 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057} 2025-10-27 11:06:13.078 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpCluster 2025-10-27 11:06:13.079 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.080 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.081 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:06:13.081 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.083 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.085 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.085 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:06:13.085 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.087 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.098 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.098 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:06:13.098 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.225 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447973225000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:06:13.225Z, endTime=2025-10-27T03:04:13.057Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:06:13.284 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 259 2025-10-27 11:06:13.285 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:06:13.309 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:06:13.309 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:06:13.378 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:06:13.378 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:06:13.404 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:06:13.404 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:06:13.410 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:06:13.449 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:06:13.225, end=2025-10-27T11:04:13.057}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:06:13.450 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.451 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.451 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:06:13.451 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.454 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.462 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.462 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:06:13.462 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.470 INFO 2458 --- [ocp-schedules-17,07e362397cc1aa37,1d2fa156d10ff24f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:13.474 INFO 2458 --- [ocp-schedules-17,07e362397cc1aa37,1d2fa156d10ff24f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:13.554 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534300, nodeCount=260, affectRows=260 2025-10-27 11:06:13.558 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534300, nodeCount=294, affectRows=294 2025-10-27 11:06:13.567 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534300, nodeCount=263, affectRows=263 2025-10-27 11:06:13.581 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761447973581000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:06:13.581Z, endTime=2025-10-27T03:04:13.057Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:06:13.596 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:06:13.596 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:06:13.624 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:06:13.624 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:06:13.633 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:06:13.633 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:06:13.657 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:06:13.657 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534253057000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:06:13.661 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:06:13.661 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:06:13.581, end=2025-10-27T11:04:13.057}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:06:13.662 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:06:13.663 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:02:12.657, end=2025-10-27T11:04:13.057},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:06:13.663 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:06:13.663 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:06:13.666 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:06:14.104 INFO 2458 --- [ocp-schedules-3,2c41a941f0a9df8a,361d7bc255b876c7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:06:14.104 INFO 2458 --- [ocp-schedules-3,2c41a941f0a9df8a,361d7bc255b876c7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:06:14.161 INFO 2458 --- [Collect all tenants major compactions1,2c41a941f0a9df8a,4e031963cbcc983e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:06:14.162 INFO 2458 --- [Collect all tenants major compactions2,2c41a941f0a9df8a,75b06fb7c4462bdf] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:06:14.162 INFO 2458 --- [Collect all tenants major compactions3,2c41a941f0a9df8a,88ab8c402ff25429] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:06:14.162 INFO 2458 --- [Collect all tenants major compactions4,2c41a941f0a9df8a,fa15161a6389fee4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:06:14.162 INFO 2458 --- [Collect all tenants major compactions5,2c41a941f0a9df8a,f4f0250ebadd104d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:06:14.162 INFO 2458 --- [Collect all tenants major compactions6,2c41a941f0a9df8a,2b04dd256a23b509] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:06:14.199 INFO 2458 --- [Collect all tenants major compactions5,2c41a941f0a9df8a,f4f0250ebadd104d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.200 INFO 2458 --- [Collect all tenants major compactions6,2c41a941f0a9df8a,2b04dd256a23b509] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.201 INFO 2458 --- [Collect all tenants major compactions5,2c41a941f0a9df8a,f4f0250ebadd104d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:06:14.202 INFO 2458 --- [Collect all tenants major compactions6,2c41a941f0a9df8a,2b04dd256a23b509] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:06:14.207 INFO 2458 --- [Collect all tenants major compactions2,2c41a941f0a9df8a,75b06fb7c4462bdf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.209 INFO 2458 --- [Collect all tenants major compactions2,2c41a941f0a9df8a,75b06fb7c4462bdf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:06:14.210 INFO 2458 --- [Collect all tenants major compactions6,2c41a941f0a9df8a,2b04dd256a23b509] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:06:14.211 INFO 2458 --- [Collect all tenants major compactions1,2c41a941f0a9df8a,4e031963cbcc983e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.211 INFO 2458 --- [Collect all tenants major compactions5,2c41a941f0a9df8a,f4f0250ebadd104d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:06:14.213 INFO 2458 --- [Collect all tenants major compactions1,2c41a941f0a9df8a,4e031963cbcc983e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:06:14.214 INFO 2458 --- [Collect all tenants major compactions3,2c41a941f0a9df8a,88ab8c402ff25429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.214 INFO 2458 --- [Collect all tenants major compactions2,2c41a941f0a9df8a,75b06fb7c4462bdf] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:06:14.216 INFO 2458 --- [Collect all tenants major compactions4,2c41a941f0a9df8a,fa15161a6389fee4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:14.216 INFO 2458 --- [Collect all tenants major compactions3,2c41a941f0a9df8a,88ab8c402ff25429] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:06:14.216 INFO 2458 --- [Collect all tenants major compactions1,2c41a941f0a9df8a,4e031963cbcc983e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:06:14.217 INFO 2458 --- [Collect all tenants major compactions4,2c41a941f0a9df8a,fa15161a6389fee4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:06:14.219 INFO 2458 --- [ocp-schedules-7,ded83dbbf38bbf50,4045462ad8d051a7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:06:14.219+08:00, clusters=[1, 2] 2025-10-27 11:06:14.219 INFO 2458 --- [Collect all tenants major compactions3,2c41a941f0a9df8a,88ab8c402ff25429] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:06:14.220 INFO 2458 --- [Collect all tenants major compactions4,2c41a941f0a9df8a,fa15161a6389fee4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:06:14.221 INFO 2458 --- [ocp-schedules-3,2c41a941f0a9df8a,361d7bc255b876c7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:06:14.229 INFO 2458 --- [ocp-schedules-7,ded83dbbf38bbf50,4045462ad8d051a7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:06:14.229 INFO 2458 --- [ocp-schedules-7,ded83dbbf38bbf50,4045462ad8d051a7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:06:14.368 INFO 2458 --- [ocp-schedules-22,0bd603e0e74ef5f4,d5077eeb27319641] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:06:14.368 INFO 2458 --- [ocp-schedules-22,0bd603e0e74ef5f4,d5077eeb27319641] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:06:14.376 INFO 2458 --- [ocp-schedules-22,0bd603e0e74ef5f4,d5077eeb27319641] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:06:14.376 INFO 2458 --- [ocp-schedules-22,0bd603e0e74ef5f4,d5077eeb27319641] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:06:14.478 INFO 2458 --- [ocp-async-6,5468644350cd0f0b,55ae187bddb583f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:14.478 WARN 2458 --- [ocp-async-5,1e1f8e441297f5af,32009bf70b4ba821] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:14.530 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=1000, affectRows=1000 2025-10-27 11:06:14.534 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=994, validCount=992 2025-10-27 11:06:14.549 ERROR 2458 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570101) Duplicate entry '835370602992181248-1761534240' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570101) Duplicate entry '835370602992181248-1761534240' for key 'PRIMARY' 2025-10-27 11:06:14.566 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=1000, affectRows=1000 2025-10-27 11:06:14.570 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=999, validCount=995 2025-10-27 11:06:14.623 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=1000, affectRows=1000 2025-10-27 11:06:14.628 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=1000, validCount=996 2025-10-27 11:06:14.669 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=1000, affectRows=1000 2025-10-27 11:06:14.674 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=996, validCount=992 2025-10-27 11:06:14.684 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=75, affectRows=75 2025-10-27 11:06:14.684 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=73, validCount=73 2025-10-27 11:06:14.707 INFO 2458 --- [prometheus-minute-schedule-1,e23d7eaa66fcff19,88d428765fc45e22] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:14.718 INFO 2458 --- [prometheus-minute-schedule-1,e23d7eaa66fcff19,88d428765fc45e22] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:14.956 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:14.957 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:15.773 INFO 2458 --- [ocp-schedules-26,e3aba34f3a489033,3c01b95f7b07e86e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:16.479 WARN 2458 --- [ocp-async-9,acc254a0a313e176,bf8aae88a0a067fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:16.479 INFO 2458 --- [ocp-async-10,7357f27494fdaf16,e5986a0ca3d5d41e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:17.461 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.468 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null 2025-10-27 11:06:17.470 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.480 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.481 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.483 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.484 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.485 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.486 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.488 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.493 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.496 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.499 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.509 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.511 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.512 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.513 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.514 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.521 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.522 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.523 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.524 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.530 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.534 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.539 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.545 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:00:17.266, end=2025-10-27T11:05:17.449} 2025-10-27 11:06:17.545 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:06:17.707 INFO 2458 --- [prometheus-minute-schedule-1,b12919e8447a40eb,21697caef6218bb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:17.720 INFO 2458 --- [prometheus-minute-schedule-1,b12919e8447a40eb,21697caef6218bb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:17.957 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:17.958 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:18.479 WARN 2458 --- [ocp-async-1,dad6432caf6d579d,851923d96ce3f41f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:18.479 INFO 2458 --- [ocp-async-7,57e46df4f0293b23,d3814a378ab7da42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:18.508 INFO 2458 --- [ocp-async-3,c61cf5c5caa36890,f539613bd9ccc3a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:18.782 INFO 2458 --- [ocp-schedules-18,8fd89ea3d8778f2f,9d7f4aca56dd2c60] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:18.783 INFO 2458 --- [ocp-schedules-18,8fd89ea3d8778f2f,9d7f4aca56dd2c60] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:06:19.000 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534300, nodeCount=1000, affectRows=1000 2025-10-27 11:06:19.004 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534300, nodeCount=996, validCount=984 2025-10-27 11:06:19.105 INFO 2458 --- [ocp-schedules-21,a17f1948ba524494,3dfccb4cee5314cd] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:06:19.428 INFO 2458 --- [ocp-schedules-4,0414d9c1bb8da3c1,d4fd400589decb89] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:06:19.460 WARN 2458 --- [scanner-112131498-1,0414d9c1bb8da3c1,baa4a5a3808cfc66] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:06:19.603 INFO 2458 --- [alarm-detect-15,0414d9c1bb8da3c1,1defe3acde8f9e10] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534345 ~ 1761534369 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534360, value=51.515)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534360, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534360=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:06:19.606 INFO 2458 --- [alarm-detect-15,0414d9c1bb8da3c1,1defe3acde8f9e10] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:06:20.479 INFO 2458 --- [ocp-async-5,4b7e5b6f35d8bf33,2432b54c1e6339a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:20.479 WARN 2458 --- [ocp-async-6,e5c1f169361ecb79,5a2b0745129a78ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:20.707 INFO 2458 --- [prometheus-minute-schedule-1,11b005fceb0f5d2f,fed753d7fedcf996] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:20.715 INFO 2458 --- [prometheus-minute-schedule-1,11b005fceb0f5d2f,fed753d7fedcf996] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:20.958 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:20.959 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:21.411 INFO 2458 --- [ocp-schedules-7,78990a740d5d8898,e1db24e54887e431] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:21.411 INFO 2458 --- [ocp-schedules-2,2a90f8242ec519cc,9b429c4743090456] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:21.411 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:06:21.411 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:06:21.411 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:06:21.412 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:06:21.412 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:06:21.412 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:06:21.412 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:06:21.412 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:06:21.413 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:06:21.413 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:06:21.428 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:21.428 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:21.429 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:21.430 INFO 2458 --- [ocp-schedules-16,6baeb293cd81d3f2,bf148b53194e9be6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:21.430 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:21.430 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:21.430 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:21.430 INFO 2458 --- [host-check-clockdiff1,6baeb293cd81d3f2,25f93862c57f98e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:06:21.431 INFO 2458 --- [host-check-clockdiff2,6baeb293cd81d3f2,bf4e67b3380013c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:06:21.431 INFO 2458 --- [host-check-clockdiff4,6baeb293cd81d3f2,b734c837cd53d54d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:06:21.431 INFO 2458 --- [host-check-clockdiff3,6baeb293cd81d3f2,5189d83e9d968781] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:06:21.431 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:21.431 INFO 2458 --- [host-check-clockdiff5,6baeb293cd81d3f2,df0438ae50ccfa91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:06:21.433 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:21.434 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:21.434 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:21.450 INFO 2458 --- [ocp-schedules-29,6ca62f5bb6f0410d,cff9d18e214a4aad] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:06:21.451 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:21.451 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:21.452 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:21.452 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:21.452 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:21.453 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:21.454 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:21.469 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:21.470 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:21.470 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:21.485 INFO 2458 --- [host-check-clockdiff4,6baeb293cd81d3f2,b734c837cd53d54d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534381 0 0 , error= 2025-10-27 11:06:21.485 INFO 2458 --- [host-check-clockdiff2,6baeb293cd81d3f2,bf4e67b3380013c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534381 -1 -1 , error= 2025-10-27 11:06:21.486 INFO 2458 --- [host-check-clockdiff3,6baeb293cd81d3f2,5189d83e9d968781] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534381 -1 -1 , error= 2025-10-27 11:06:21.486 INFO 2458 --- [host-check-clockdiff1,6baeb293cd81d3f2,25f93862c57f98e8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534381 0 0 , error= 2025-10-27 11:06:21.504 INFO 2458 --- [host-check-clockdiff5,6baeb293cd81d3f2,df0438ae50ccfa91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534381 0 0 , error= 2025-10-27 11:06:21.546 INFO 2458 --- [ocp-schedules-18,10e59cc254636f72,8e2db642fd27a475] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:06:21.546 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:06:21.546 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:06:21.547 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:06:21.548 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:06:21.547151474+08:00 2025-10-27 11:06:21.548 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:06:21.547442998+08:00 2025-10-27 11:06:21.548 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:06:21.549 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:06:21.548+08:00 2025-10-27 11:06:21.549 INFO 2458 --- [ocp-schedules-28,fc965d6c88607cb2,934e39f8860f5603] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:06:21.549 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:06:21.549138601+08:00 2025-10-27 11:06:21.549 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:06:21.548+08:00 2025-10-27 11:06:21.549 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:06:21.549+08:00 2025-10-27 11:06:21.550 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:06:21.550 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:06:21.552 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:06:21.552 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:06:21.552 INFO 2458 --- [host-check-agent-status4,2a90f8242ec519cc,56110da1c17d9491] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:06:21.552 INFO 2458 --- [ocp-schedules-18,10e59cc254636f72,8e2db642fd27a475] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:21.552 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:06:21.552672711+08:00 2025-10-27 11:06:21.552 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:06:21.551817481+08:00 2025-10-27 11:06:21.553 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:06:21.552+08:00 2025-10-27 11:06:21.553 INFO 2458 --- [ocp-schedules-28,fc965d6c88607cb2,934e39f8860f5603] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:21.553 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:06:21.552+08:00 2025-10-27 11:06:21.554 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:06:21.554 INFO 2458 --- [ocp-schedules-12,ec756908198bef23,8b61a6d0bba7b762] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:06:21.554 INFO 2458 --- [ocp-schedules-28,fc965d6c88607cb2,934e39f8860f5603] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:06:21.555 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:06:21.556 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:06:21.556 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:06:21.556 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:06:21.556 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:06:21.556 INFO 2458 --- [host-check-agent-status2,2a90f8242ec519cc,d5943e64affb587c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:06:21.557 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:06:21.548+08:00 and status AVAILABLE 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status5,2a90f8242ec519cc,89b1dde6b4a40b74] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:06:21.558 INFO 2458 --- [host-check-agent-status3,2a90f8242ec519cc,f51a9e00f5de3b0c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:06:21.559 INFO 2458 --- [ocp-schedules-12,ec756908198bef23,8b61a6d0bba7b762] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:21.559 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:06:21.548+08:00 and status AVAILABLE 2025-10-27 11:06:21.560 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:06:21.561 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:06:21.561 INFO 2458 --- [host-check-agent-status1,2a90f8242ec519cc,74d97c8ac43f033f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:06:21.561 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:06:21.549+08:00 and status AVAILABLE 2025-10-27 11:06:21.562 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:06:21.552+08:00 and status AVAILABLE 2025-10-27 11:06:21.562 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:06:21.552+08:00 and status AVAILABLE 2025-10-27 11:06:21.563 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:06:21.567 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:06:21.568 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:06:21.568 INFO 2458 --- [ocp-schedules-26,eef1f95d38cffd50,9e5ceb955887b16b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:06:21.572 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:06:21.572 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:06:21.597 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:21.599 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:21.600 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:21.600 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:21.603 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:21.603 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:06:21.603 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:21.603 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:21.607 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:21.607 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:21.646 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.647 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.647 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.647 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.647 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.648 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:21.653 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:06:21.653 INFO 2458 --- [ocp-schedules-18,10e59cc254636f72,8e2db642fd27a475] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:06:21.664 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.665 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:21.687 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:21.692 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:21.695 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:21.697 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:21.698 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:21.707 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:06:21.711 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:06:21.713 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:21.712331+08:00, lastProcessTime=2025-10-27T11:06:11.717466+08:00 2025-10-27 11:06:21.716 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:06:21.717 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:06:21.721 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.721 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.723 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.723 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:21.723 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:06:21.744 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:21.745 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.745 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:06:21.751 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:06:21.754 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:06:21.757 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:06:21.758 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:06:21.761 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:06:21.785 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:21.787 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:21.792 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:21.793 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:21.793 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:21.802 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:06:21.804 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:06:21.809 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:06:21.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:21.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:21.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.812 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.812 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.813 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:06:21.813 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:21.813 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.813 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.813 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.814 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:06:21.814 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:06:21.819 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:21.829 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:06:21.842 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:06:21.842 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:21.842 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:21.847 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:06:21.850 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:06:21.860 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:06:21.861 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:06:21.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:21.871 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:21.881 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:21.882 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:21.888 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:06:21.889 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.889 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:21.889 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:21.890 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:21.890 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:06:21.898 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:06:21.903 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:06:21.905 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:06:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.915 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:06:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:21.917 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:21.917 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:21.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:21.921 INFO 2458 --- [host-check-host-status4,78990a740d5d8898,ad278fb5c3b8031d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:06:21.921 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:21.922 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:06:21.923 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:06:21.924 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:21.934 INFO 2458 --- [host-check-host-status2,78990a740d5d8898,807e7277e01cc29c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:06:21.938 INFO 2458 --- [host-check-host-status5,78990a740d5d8898,5f4df4714b0e4e52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:06:21.955 INFO 2458 --- [host-check-host-status3,78990a740d5d8898,7055ce5d7d580e51] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:06:21.960 INFO 2458 --- [host-check-host-status1,78990a740d5d8898,7651ce70bc88da73] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:06:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:21.986 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:06:22.006 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:21.712331+08:00 2025-10-27 11:06:22.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:22.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:22.029 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:06:22.030 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.030 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:06:22.032 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:06:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:22.050 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:06:22.072 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:22.073 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.073 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:06:22.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:22.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:22.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:06:22.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:06:22.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:22.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.206 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:22.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:06:22.220 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:06:22.229 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:22.232 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:22.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:06:22.333 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:06:22.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.333 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.333 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.334 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:06:22.336 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.336 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.336 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.337 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:06:22.375 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:06:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:22.417 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:22.417 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:06:22.457 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:06:22.457 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:06:22.458 INFO 2458 --- [ocp-schedules-12,ec756908198bef23,8b61a6d0bba7b762] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:06:22.479 WARN 2458 --- [ocp-async-4,a4e54911c038c9bd,1f3fa016589ea5f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:22.479 INFO 2458 --- [ocp-async-10,eb3f92d5a85f446d,04468e3cabecaa73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:23.484 INFO 2458 --- [ocp-schedules-4,4e524d393c2c0957,a73b744cdd543968] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:23.488 INFO 2458 --- [ocp-schedules-4,4e524d393c2c0957,a73b744cdd543968] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:23.707 INFO 2458 --- [prometheus-minute-schedule-1,ef5973000c6ec031,22da2b082971bee9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:23.715 INFO 2458 --- [prometheus-minute-schedule-1,ef5973000c6ec031,22da2b082971bee9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:23.959 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:23.959 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:24.479 WARN 2458 --- [ocp-async-1,f9c7a94a32783241,e0807ffda979d60c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:24.479 INFO 2458 --- [ocp-async-1,266bd8182688d7b5,4689fc4fbbea6c8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:25.785 INFO 2458 --- [ocp-schedules-11,9159e435b9cf6c3b,05adfcf448ab74f0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:26.479 WARN 2458 --- [ocp-async-3,4d7cff806c21ce86,07ce52179bcc4f49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:26.479 INFO 2458 --- [ocp-async-3,02fd8fcc07e2ac20,1c4566d88e43f760] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:26.645 INFO 2458 --- [ocp-schedules-25,215aaeae16d89ff9,859afea98709a7a1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:06:26.652 INFO 2458 --- [ocp-schedules-25,215aaeae16d89ff9,859afea98709a7a1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:26.652 INFO 2458 --- [ocp-schedules-25,215aaeae16d89ff9,859afea98709a7a1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:26.707 INFO 2458 --- [prometheus-minute-schedule-1,d8615d54a8c40710,0ea5464704400418] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:26.715 INFO 2458 --- [prometheus-minute-schedule-1,d8615d54a8c40710,0ea5464704400418] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:26.960 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:26.961 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:28.479 INFO 2458 --- [ocp-async-5,5e74f7a0213b2540,82a03e7ac00b9dce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:28.479 WARN 2458 --- [ocp-async-4,223cc04ca45ad3bd,58afefea405ef178] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:28.508 INFO 2458 --- [ocp-async-10,69a50faa48dce2f8,d84fc83b494f881f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:28.837 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:06:28.837 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:06:28.839 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:06:28.839 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839256866908741632 2025-10-27 11:06:28.840 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=1, maxSyncedRecordId=839258596677459968 2025-10-27 11:06:28.841 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:06:28.841 INFO 2458 --- [ocp-schedules-7,737081eec9892c76,9cffc4579b84fa02] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:06:29.302 INFO 2458 --- [ocp-schedules-20,a9f7706699dfb407,e8b788761d97beb2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:06:29.307 INFO 2458 --- [ocp-schedules-20,a9f7706699dfb407,e8b788761d97beb2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:06:29.307 INFO 2458 --- [ocp-schedules-20,a9f7706699dfb407,e8b788761d97beb2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:06:29.637 INFO 2458 --- [ocp-schedules-5,e8039a42c395a2dd,2c33e1437c577dfe] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:06:29.698 WARN 2458 --- [scanner-112131684-1,e8039a42c395a2dd,77343c3788f58319] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:06:29.707 INFO 2458 --- [prometheus-minute-schedule-1,8a715dd504fd767c,d10662d953cff986] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:29.719 INFO 2458 --- [prometheus-minute-schedule-1,8a715dd504fd767c,d10662d953cff986] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:29.829 INFO 2458 --- [alarm-detect-2,e8039a42c395a2dd,b5e2f766e4f7cea8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534360 ~ 1761534379 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534375, value=57.208)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534375, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534375=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:06:29.961 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:29.961 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:30.480 WARN 2458 --- [ocp-async-5,f765103697493803,1d786d52b94be5ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:30.480 INFO 2458 --- [ocp-async-5,06905d77c1571d52,1eb240aa539fcd77] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:31.714630+08:00, lastProcessTime=2025-10-27T11:06:21.712331+08:00 2025-10-27 11:06:31.915 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:31.714630+08:00 2025-10-27 11:06:32.480 WARN 2458 --- [ocp-async-5,66bd1c447e92b6f8,a5cad616d44770f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:32.480 INFO 2458 --- [ocp-async-6,f1b34cfa3ce2afcf,d375a8ad4c1f1e0c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:32.707 INFO 2458 --- [prometheus-minute-schedule-1,261062e3f8123a92,25b41e48c9a480fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:32.715 INFO 2458 --- [prometheus-minute-schedule-1,261062e3f8123a92,25b41e48c9a480fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:32.962 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:32.963 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:33.499 INFO 2458 --- [ocp-schedules-1,66365c007e22666d,8223c39001965070] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:33.503 INFO 2458 --- [ocp-schedules-1,66365c007e22666d,8223c39001965070] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:33.727 INFO 2458 --- [ocp-schedules-23,199999da93138091,81592beaca13d90e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:06:33.727 INFO 2458 --- [ocp-schedules-23,199999da93138091,81592beaca13d90e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:06:33.922 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:06:33.922 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:06:33.944 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:06:33.944 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:06:33.944 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:06:33.959 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:06:33.959 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:06:33.959 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:06:33.972 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:06:33.972 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:06:33.972 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:06:33.983 INFO 2458 --- [ocp-schedules-24,dc66d0b77a4274b6,bcc1c3ff438ab127] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:06:34.480 WARN 2458 --- [ocp-async-9,73537fa922f214fa,e358556ac78f37ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:34.480 INFO 2458 --- [ocp-async-6,6780d915b4d399f3,f4b7926fd8b59fd7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:34.712 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:06:34.712 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:06:34.787 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:34.787 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:34.824 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:34.824 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:34.825 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:34.826 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:06:34.840 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:34.840 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:34.840 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:34.841 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:06:34.907 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:34.908 INFO 2458 --- [ocp-schedules-26,f1f917ff8d110b8b,f922c9e9a2dcc7f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:06:35.641 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-10-27 11:06:35.641 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.641 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-10-27 11:06:35.641 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-10-27 11:06:35.641 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-10-27 11:06:35.642 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-10-27 11:06:35.643 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ssl_certificate_is_expired, expression ssl_certificate_expired_status > 0 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ssl_certificate_expired_status', labels={}}], alarmTargets 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ssl_certificate_is_expired 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ssl_certificate_is_expired 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-10-27 11:06:35.644 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-10-27 11:06:35.645 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-10-27 11:06:35.646 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-10-27 11:06:35.647 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-10-27 11:06:35.648 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-10-27 11:06:35.649 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-10-27 11:06:35.650 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-10-27 11:06:35.651 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-10-27 11:06:35.652 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-10-27 11:06:35.653 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 10 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 10 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-10-27 11:06:35.654 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-10-27 11:06:35.655 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 10 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-10-27 11:06:35.656 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-10-27 11:06:35.657 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-10-27 11:06:35.658 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 10 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.659 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-10-27 11:06:35.660 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 10 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-10-27 11:06:35.661 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-10-27 11:06:35.662 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-10-27 11:06:35.663 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-10-27 11:06:35.664 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-10-27 11:06:35.665 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-10-27 11:06:35.666 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-10-27 11:06:35.667 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 10 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-10-27 11:06:35.668 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-10-27 11:06:35.669 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-10-27 11:06:35.670 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-10-27 11:06:35.671 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration == 0 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration > 0 and ob_license_expired_second_duration < 2592000 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration >= 2592000 and ob_license_expired_second_duration < 5184000 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_license_expired_second_duration', labels={}}], alarmTargets 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_license_expired 2025-10-27 11:06:35.672 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_license_expired 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-10-27 11:06:35.673 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-10-27 11:06:35.674 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-10-27 11:06:35.675 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-10-27 11:06:35.676 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.677 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-10-27 11:06:35.678 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-10-27 11:06:35.679 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-10-27 11:06:35.680 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.681 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-10-27 11:06:35.682 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-10-27 11:06:35.683 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-10-27 11:06:35.684 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-10-27 11:06:35.685 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-10-27 11:06:35.686 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-10-27 11:06:35.687 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:06:35.687 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-10-27 11:06:35.687 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-10-27 11:06:35.687 INFO 2458 --- [ocp-schedules-12,b31e7ab679dc50d8,9df51e45812c7a32] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=134, updatedCount=134, failedToUpdateCount=0 2025-10-27 11:06:35.707 INFO 2458 --- [prometheus-minute-schedule-1,15759a92e4bda9f4,22c49bc37881d6af] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:35.715 INFO 2458 --- [prometheus-minute-schedule-1,15759a92e4bda9f4,22c49bc37881d6af] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:35.794 INFO 2458 --- [ocp-schedules-15,701057df1178bb24,f2c0463929b1cbf0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:35.963 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:35.964 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:36.480 WARN 2458 --- [ocp-async-6,79ff7333326ac1f7,8e6c3f84f391ce04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:36.480 INFO 2458 --- [ocp-async-8,91d199f25d69038c,56a055fb452b0e75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:36.868 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:36.868 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:06:36.869 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:06:36.869 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:06:36.869 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:06:36.869 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:06:36.880 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:36.883 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:36.883 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:36.884 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:36.885 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:36.885 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:36.887 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:36.889 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:36.889 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:36.889 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:36.943 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:06:36.944 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:06:36.944 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:06:36.947 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:06:36.948 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:06:36.950 INFO 2458 --- [ocp-schedules-30,43981fd7e12eaad0,a07de4067234042e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:06:37.015 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,2eaad9a272b9262f,c0441646d27efa3e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=2eaad9a272b9262f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:06:37.016 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,2eaad9a272b9262f,c0441646d27efa3e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=2eaad9a272b9262f, duration=1 ms] 2025-10-27 11:06:37.708 INFO 2458 --- [prometheus-cache-monitor1,7d4596f34c5c649f,adcfa5dfdc940e10] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20298,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5080,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7861} 2025-10-27 11:06:37.708 INFO 2458 --- [prometheus-cache-monitor1,7d4596f34c5c649f,adcfa5dfdc940e10] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:06:38.480 WARN 2458 --- [ocp-async-8,13d342d9175df201,a9dbdaeead676ec4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:38.480 INFO 2458 --- [ocp-async-3,8451dd9fc803dcb4,76676dac5990db1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:38.508 INFO 2458 --- [ocp-async-9,1b497f44805e57d3,15de5456a4a5b3a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:38.578 INFO 2458 --- [ocp-schedules-21,252689feca7ebeb9,2412bb0d46f3f441] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:06:38.583 INFO 2458 --- [ocp-schedules-21,252689feca7ebeb9,2412bb0d46f3f441] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:38.583 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:06:38.584 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:06:38.584 INFO 2458 --- [ocp-schedules-21,252689feca7ebeb9,2412bb0d46f3f441] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:06:38.707 INFO 2458 --- [prometheus-minute-schedule-1,8c93e60354a68da0,c7f91543e17373ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:38.713 INFO 2458 --- [prometheus-minute-schedule-1,8c93e60354a68da0,c7f91543e17373ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:38.964 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:38.965 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:39.834 INFO 2458 --- [ocp-schedules-28,79ef3735ab32ad24,c99466bd8324a041] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:06:39.904 WARN 2458 --- [scanner-112131930-1,79ef3735ab32ad24,c16258a1b8e05ae5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:06:40.051 INFO 2458 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:06:40.051 INFO 2458 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:06:40.064 INFO 2458 --- [alarm-detect-1,79ef3735ab32ad24,813778644dd03610] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534269 ~ 1761534329 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=202634.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202634.56, values={1761534300=202634.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:06:40.064 INFO 2458 --- [alarm-detect-1,79ef3735ab32ad24,813778644dd03610] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_compaction, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:06:40.064 INFO 2458 --- [alarm-detect-1,79ef3735ab32ad24,813778644dd03610] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:06:40.106 INFO 2458 --- [Check obproxy1,305721cecf95acd2,c54ad9eb55a17707] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:06:40.106 INFO 2458 --- [Check obproxy1,305721cecf95acd2,c54ad9eb55a17707] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:06:40.106 INFO 2458 --- [Collect obproxy1,9596785dc09fd5cb,dbcb6cee098dc711] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:06:40.106 INFO 2458 --- [Collect obproxy1,9596785dc09fd5cb,dbcb6cee098dc711] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:06:40.107 INFO 2458 --- [alarm-detect-20,79ef3735ab32ad24,54dc67e37a8c768e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534270 ~ 1761534330 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=92.951)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=92.951, values={1761534300=92.951}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:06:40.107 INFO 2458 --- [alarm-detect-20,79ef3735ab32ad24,54dc67e37a8c768e] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_log_disk_usage_high, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:06:40.108 INFO 2458 --- [alarm-detect-20,79ef3735ab32ad24,54dc67e37a8c768e] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:06:40.136 INFO 2458 --- [Collect obproxy1,9596785dc09fd5cb,dbcb6cee098dc711] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:06:40.141 INFO 2458 --- [Collect obproxy1,9596785dc09fd5cb,dbcb6cee098dc711] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:06:40.145 INFO 2458 --- [Collect obproxy1,9596785dc09fd5cb,dbcb6cee098dc711] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:06:40.155 INFO 2458 --- [Check obproxy1,305721cecf95acd2,c54ad9eb55a17707] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:06:40.158 INFO 2458 --- [Check obproxy1,305721cecf95acd2,c54ad9eb55a17707] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:06:40.158 INFO 2458 --- [Check obproxy1,305721cecf95acd2,c54ad9eb55a17707] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:06:40.282 INFO 2458 --- [alarm-detect-17,79ef3735ab32ad24,0e844f2aeaf0b59f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534270 ~ 1761534330 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534300, value=202635.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202635.912, values={1761534300=202635.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:06:40.283 INFO 2458 --- [alarm-detect-17,79ef3735ab32ad24,0e844f2aeaf0b59f] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_frozen, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:06:40.283 INFO 2458 --- [alarm-detect-17,79ef3735ab32ad24,0e844f2aeaf0b59f] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:06:40.290 INFO 2458 --- [alarm-detect-10,79ef3735ab32ad24,d1485c59c07675b1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534360 ~ 1761534390 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534390, value=60.961), ValueData(timestamp=1761534375, value=57.208)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534390, value=17409.0), ValueData(timestamp=1761534375, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534390=17409.0, 1761534375=17409.0}, valueCount=2, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:06:40.303 INFO 2458 --- [ocp-schedules-11,2dc41440c2004360,78879a058f056fe3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:06:40.303 INFO 2458 --- [ocp-schedules-11,2dc41440c2004360,78879a058f056fe3] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:06:40.310 INFO 2458 --- [alarm-detect-10,79ef3735ab32ad24,d1485c59c07675b1] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:06:40.329 INFO 2458 --- [ocp-schedules-11,2dc41440c2004360,78879a058f056fe3] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:06:40.480 WARN 2458 --- [ocp-async-4,f84f249a786ae074,ea38abb4b96f1659] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:40.480 INFO 2458 --- [ocp-async-7,0667afa2d638021b,130b81d47592b99b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:41.707 INFO 2458 --- [prometheus-minute-schedule-1,b33f71fc392b692b,f3a2e74ab66b542a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:41.713287+08:00, lastProcessTime=2025-10-27T11:06:31.714630+08:00 2025-10-27 11:06:41.715 INFO 2458 --- [prometheus-minute-schedule-1,b33f71fc392b692b,f3a2e74ab66b542a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:41.945 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:41.713287+08:00 2025-10-27 11:06:41.965 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:41.966 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:42.480 INFO 2458 --- [ocp-async-1,01cd9cdde1786f8c,b049931b2673380b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:42.481 WARN 2458 --- [ocp-async-6,2c06dc507555db22,a5466fe3bf22fc9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:43.512 INFO 2458 --- [ocp-schedules-4,da4a475fa2990c65,4f45dce2585e646a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:43.516 INFO 2458 --- [ocp-schedules-4,da4a475fa2990c65,4f45dce2585e646a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:43.973 INFO 2458 --- [ocp-schedules-17,c68082d412690009,550b1d66b85684ff] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:06:44.481 WARN 2458 --- [ocp-async-5,e21875758f06aeb9,9f8e4164a52bc7e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:44.481 INFO 2458 --- [ocp-async-2,e33150b9344097ca,1818862a821d0a7d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:44.548 INFO 2458 --- [ocp-schedules-20,4eb774191da8ecd8,516236d753ebda20] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:06:44.549 INFO 2458 --- [ocp-schedules-20,4eb774191da8ecd8,516236d753ebda20] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:06:44.556 INFO 2458 --- [ocp-schedules-20,4eb774191da8ecd8,516236d753ebda20] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:06:44.556 INFO 2458 --- [ocp-schedules-20,4eb774191da8ecd8,516236d753ebda20] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:06:44.707 INFO 2458 --- [prometheus-minute-schedule-1,6ffa3e4feecd602c,a1378a69deb9aabe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:44.715 INFO 2458 --- [prometheus-minute-schedule-1,6ffa3e4feecd602c,a1378a69deb9aabe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:44.966 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:44.967 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:45.102 INFO 2458 --- [prometheus-second-schedule-1,b51dc693a4f5c6b7,9bcf23d45945b692] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:06:45.801 INFO 2458 --- [ocp-schedules-4,e0575e1e7a00c0c2,e4be324283c0d861] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:46.481 INFO 2458 --- [ocp-async-3,5bef283d1ab28824,7d890b19575b1657] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:46.481 WARN 2458 --- [ocp-async-4,cc9abb1e0106ade6,ca93e14eac2a6149] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:46.655 INFO 2458 --- [ocp-schedules-15,272f9bcd50a68ee0,35f3a81e863cda44] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:06:46.663 INFO 2458 --- [ocp-schedules-15,272f9bcd50a68ee0,35f3a81e863cda44] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:46.663 INFO 2458 --- [ocp-schedules-15,272f9bcd50a68ee0,35f3a81e863cda44] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:06:47.657 INFO 2458 --- [ocp-schedules-24,7232a922bbefdecb,95db79923f5b9995] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:06:47.657+08:00, clusters=[1, 2] 2025-10-27 11:06:47.672 INFO 2458 --- [ocp-schedules-24,7232a922bbefdecb,95db79923f5b9995] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:06:47.675 INFO 2458 --- [ocp-schedules-24,7232a922bbefdecb,95db79923f5b9995] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:06:47.707 INFO 2458 --- [prometheus-minute-schedule-1,62f06917102215e3,ec9ecf6c12ed1eae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:47.714 INFO 2458 --- [prometheus-minute-schedule-1,62f06917102215e3,ec9ecf6c12ed1eae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:47.967 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:47.968 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:48.481 WARN 2458 --- [ocp-async-7,2846186cae49d260,6251d1e5bcc70650] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:48.481 INFO 2458 --- [ocp-async-6,d132847a7619c37f,b7cb25dbaf696a5f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:48.508 INFO 2458 --- [ocp-async-1,b1d1f67c7c4bfa5d,50269a5c795c401e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:50.312 INFO 2458 --- [ocp-schedules-24,0b59af48f567a44e,cfa6eb0891e6effd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:06:50.458 WARN 2458 --- [scanner-112132216-1,0b59af48f567a44e,c4d28a35ebe6f0e5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:06:50.481 INFO 2458 --- [ocp-async-5,b1429626bb090306,c18e49608546ff74] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:50.482 WARN 2458 --- [ocp-async-2,3b6ac033fe580e16,4ffa6420320c6c28] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:50.707 INFO 2458 --- [prometheus-minute-schedule-1,004b6f9129c0416e,245708bc9558fa43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:50.714 INFO 2458 --- [prometheus-minute-schedule-1,004b6f9129c0416e,245708bc9558fa43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:50.968 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:50.969 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:51.409 INFO 2458 --- [ocp-schedules-20,55f21c07e1649395,bd072f8d8ad6aead] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:51.409 INFO 2458 --- [ocp-schedules-6,f58845ab11e818b8,cef124d1f90817eb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:06:51.410 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:06:51.411 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:06:51.411 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:06:51.411 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:06:51.411 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:06:51.425 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:51.425 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:51.425 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:51.425 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:51.426 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:51.427 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:51.427 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:51.430 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:51.431 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:51.431 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:51.431 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:51.431 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:51.432 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:51.432 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:51.432 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:51.432 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:51.433 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:51.433 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:51.434 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:51.442 INFO 2458 --- [ocp-schedules-11,38e605760ce5a872,ee1ed3b14b05ff8d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:06:51.443 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:51.447 INFO 2458 --- [ocp-schedules-13,786a5896e5ba748c,c5dd3b187a1e3020] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:06:51.448 INFO 2458 --- [host-check-clockdiff1,786a5896e5ba748c,244cc7e7bb2047d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:06:51.448 INFO 2458 --- [host-check-clockdiff2,786a5896e5ba748c,77d58c6d59c91f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:06:51.448 INFO 2458 --- [host-check-clockdiff4,786a5896e5ba748c,4a39f5878f85c6e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:06:51.448 INFO 2458 --- [host-check-clockdiff3,786a5896e5ba748c,45a269063b0d1e8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:06:51.448 INFO 2458 --- [host-check-clockdiff5,786a5896e5ba748c,36bfbfe559ebc5e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:06:51.460 INFO 2458 --- [host-check-clockdiff1,786a5896e5ba748c,244cc7e7bb2047d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534411 0 0 , error= 2025-10-27 11:06:51.464 INFO 2458 --- [host-check-clockdiff3,786a5896e5ba748c,45a269063b0d1e8a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534411 -1 -1 , error= 2025-10-27 11:06:51.480 INFO 2458 --- [host-check-clockdiff5,786a5896e5ba748c,36bfbfe559ebc5e7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534411 0 0 , error= 2025-10-27 11:06:51.516 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:06:51.517 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:06:51.518 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:06:51.519 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:06:51.518908106+08:00 2025-10-27 11:06:51.520 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:06:51.520034939+08:00 2025-10-27 11:06:51.520 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:06:51.520 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:06:51.520 INFO 2458 --- [host-check-agent-status2,55f21c07e1649395,53c135e56e34823f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:06:51.521 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:06:51.520+08:00 2025-10-27 11:06:51.521 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:06:51.519+08:00 2025-10-27 11:06:51.522 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:06:51.523 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:06:51.523 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:06:51.523 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:06:51.523 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:06:51.522637174+08:00 2025-10-27 11:06:51.524 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:06:51.524 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:06:51.523+08:00 2025-10-27 11:06:51.524 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:06:51.525 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:06:51.525 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:06:51.525 INFO 2458 --- [host-check-agent-status3,55f21c07e1649395,ef1e667c03b148c4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:06:51.526 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:06:51.525317216+08:00 2025-10-27 11:06:51.526 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:06:51.526 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:06:51.526 INFO 2458 --- [host-check-agent-status1,55f21c07e1649395,479d7dac16e60640] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:06:51.526 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:06:51.526+08:00 2025-10-27 11:06:51.527 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:06:51.527 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:06:51.527 INFO 2458 --- [host-check-agent-status4,55f21c07e1649395,8d7ddcbba37886c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:06:51.527 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:06:51.528 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:06:51.528 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:06:51.528 INFO 2458 --- [host-check-agent-status5,55f21c07e1649395,0a136924e542654d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:06:51.528 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:06:51.519+08:00 and status AVAILABLE 2025-10-27 11:06:51.529 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:06:51.520+08:00 and status AVAILABLE 2025-10-27 11:06:51.529 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:06:51.529630378+08:00 2025-10-27 11:06:51.530 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:06:51.529+08:00 2025-10-27 11:06:51.532 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:06:51.523+08:00 and status AVAILABLE 2025-10-27 11:06:51.534 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:06:51.535 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:06:51.526+08:00 and status AVAILABLE 2025-10-27 11:06:51.536 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:06:51.537 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:06:51.529+08:00 and status AVAILABLE 2025-10-27 11:06:51.538 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:06:51.541 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:06:51.544 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:06:51.545 INFO 2458 --- [ocp-schedules-19,a3fb6115eaad7e66,427a63dd0a3a2bd6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:06:51.549 INFO 2458 --- [ocp-schedules-25,167a4bc7529caa5c,f7543e89a5c301ae] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:06:51.551 INFO 2458 --- [ocp-schedules-19,a3fb6115eaad7e66,427a63dd0a3a2bd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:51.554 INFO 2458 --- [ocp-schedules-29,4a350ef816ff1c9d,35666a111aac4b96] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:06:51.556 INFO 2458 --- [ocp-schedules-25,167a4bc7529caa5c,f7543e89a5c301ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:51.556 INFO 2458 --- [ocp-schedules-25,167a4bc7529caa5c,f7543e89a5c301ae] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:06:51.558 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:06:51.558 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:06:51.559 INFO 2458 --- [ocp-schedules-29,4a350ef816ff1c9d,35666a111aac4b96] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:51.560 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:51.561 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:06:51.562 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:06:51.563 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:51.564 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:06:51.565 INFO 2458 --- [ocp-schedules-14,4d066ac2790f0a48,1805c2440f72195e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:06:51.567 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:06:51.569 INFO 2458 --- [ocp-schedules-5,9b1e90b6958a7314,069156e6700af281] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:06:51.570 INFO 2458 --- [ocp-schedules-10,9724f4d1e2f64ef8,88c1a49af5a4b2cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:06:51.572 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:51.577 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:51.579 INFO 2458 --- [ocp-schedules-10,9724f4d1e2f64ef8,88c1a49af5a4b2cc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:06:51.580 INFO 2458 --- [ocp-schedules-5,9b1e90b6958a7314,069156e6700af281] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:06:51.604 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.605 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:51.606 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:06:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:51.626 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:06:51.628 INFO 2458 --- [ocp-schedules-19,a3fb6115eaad7e66,427a63dd0a3a2bd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:06:51.630 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:51.630 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:51.633 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:51.637 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.637 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:51.639 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:06:51.648 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:06:51.649 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:06:51.649 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:06:51.656 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:06:51.662 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:06:51.680 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:06:51.686 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.686 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.686 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.686 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.686 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.686 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:06:51.686 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:06:51.687 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:51.695 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:06:51.698 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:06:51.700 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:51.701 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.701 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:06:51.719 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:51.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:06:51.720382+08:00, lastProcessTime=2025-10-27T11:06:41.713287+08:00 2025-10-27 11:06:51.729 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:51.730 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:51.731 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:51.736 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:06:51.741 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:06:51.747 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:06:51.748 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:06:51.752 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:06:51.766 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:06:51.771 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:51.772 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.772 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:51.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:06:51.777 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:51.785 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.791 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:06:51.792 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:06:51.796 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:06:51.797 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:06:51.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:51.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:51.804 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:06:51.823 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:51.827 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:51.827 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:51.830 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:51.834 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:06:51.842 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:06:51.843 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:06:51.844 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:06:51.849 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:06:51.878 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:06:51.879 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:51.881 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.882 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:51.883 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:06:51.895 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:06:51.904 INFO 2458 --- [host-check-host-status2,f58845ab11e818b8,5cc3730cfb18ce41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:06:51.904 INFO 2458 --- [host-check-host-status5,f58845ab11e818b8,3537abd31fa59fc6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:06:51.905 INFO 2458 --- [host-check-host-status3,f58845ab11e818b8,eea6ba6d6a018fbd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:06:51.907 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:06:51.910 INFO 2458 --- [host-check-host-status4,f58845ab11e818b8,97e307446e9ee922] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:06:51.910 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:06:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.913 INFO 2458 --- [host-check-host-status1,f58845ab11e818b8,9b6c5150c8b767c7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:06:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:06:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:51.919 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:06:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:51.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:51.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:51.970 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:06:52.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:52.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:52.012 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:06:52.012 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.013 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:06:52.015 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.016 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:06:52.032 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:06:51.720382+08:00 2025-10-27 11:06:52.033 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:06:52.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:52.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.057 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:06:52.081 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:06:52.081 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:06:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:06:52.124 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:52.124 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:06:52.167 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:06:52.207 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:52.207 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.207 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.208 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.208 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.208 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:06:52.212 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:06:52.223 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:06:52.232 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:06:52.235 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:06:52.289 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:06:52.344 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:06:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:06:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:06:52.384 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:06:52.427 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:52.427 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.427 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.428 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:52.428 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:52.428 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:52.429 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:06:52.470 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:06:52.470 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:06:52.470 INFO 2458 --- [ocp-schedules-29,4a350ef816ff1c9d,35666a111aac4b96] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:06:52.481 INFO 2458 --- [ocp-async-9,c3c0ba0d5e2f13e5,b1e7d99e556f4341] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:52.481 WARN 2458 --- [ocp-async-10,4befe57cfc316f67,e8ff4a89e60e62a5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:06:53.246 INFO 2458 --- [host-check-clockdiff4,786a5896e5ba748c,4a39f5878f85c6e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534413 0 0 , error= 2025-10-27 11:06:53.245 INFO 2458 --- [host-check-clockdiff2,786a5896e5ba748c,77d58c6d59c91f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534413 -1 -1 , error= 2025-10-27 11:06:53.524 INFO 2458 --- [ocp-schedules-3,e09486afe8c34467,a6badae4d8b53fee] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:06:53.527 INFO 2458 --- [ocp-schedules-3,e09486afe8c34467,a6badae4d8b53fee] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:06:53.707 INFO 2458 --- [prometheus-minute-schedule-1,e6abac92d902daae,459ab260d110a660] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:53.715 INFO 2458 --- [prometheus-minute-schedule-1,e6abac92d902daae,459ab260d110a660] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:53.969 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:53.970 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:53.980 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,3616c874b332741a,7c56ae38de4f7c8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=3616c874b332741a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:06:53.980 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,3616c874b332741a,7c56ae38de4f7c8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=3616c874b332741a, duration=1 ms] 2025-10-27 11:06:54.373 INFO 2458 --- [ocp-schedules-11,fd5a1168da970821,73e47d1ac2b79912] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:06:54.482 INFO 2458 --- [ocp-async-9,507b5a233faf2c22,081b3cfc831cbca5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:54.483 WARN 2458 --- [ocp-async-1,28380ff5e00fa6c2,b6ea9aceac5f0036] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:55.808 INFO 2458 --- [ocp-schedules-4,8889f27451415208,739da46a7c441764] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:06:56.482 WARN 2458 --- [ocp-async-8,b0bb39878510dd9c,f636fc0f2ad47b6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:56.483 INFO 2458 --- [ocp-async-8,849bd8b9132de178,217ab532c8fc8025] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:56.707 INFO 2458 --- [prometheus-minute-schedule-1,2e9058235202e306,6d8f298a1ef3e8de] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:56.714 INFO 2458 --- [prometheus-minute-schedule-1,2e9058235202e306,6d8f298a1ef3e8de] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:56.970 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:56.971 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:06:57.026 INFO 2458 --- [ocp-schedules-19,1a4c087e6a6b99bb,9c417a4a25a837f7] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:06:57.033 INFO 2458 --- [ocp-schedules-19,1a4c087e6a6b99bb,9c417a4a25a837f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:06:57.119 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:06:57.120 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.123 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:06:57.131 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.132 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.157 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.158 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.167 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.167 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.168 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:06:57.168 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.168 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.168 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.173 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:06:57.186 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.186 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.188 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.188 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.188 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.189 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.189 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.189 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.212 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.213 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.223 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.223 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.224 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.224 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.235 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.236 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:06:57.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.262 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.263 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.263 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.264 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.265 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.266 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.267 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.269 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.270 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.270 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.271 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.272 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.272 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:06:57.275 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:06:57.285 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.285 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.285 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.286 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.314 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.314 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:06:57.316 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.316 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.316 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.317 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:06:57.335 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:57.336 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.336 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:06:57.346 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.346 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:06:57.346 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.346 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:06:57.346 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.347 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.347 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:06:57.349 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:06:57.349 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:06:57.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.377 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.377 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:06:57.380 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.380 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.380 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.380 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:06:57.409 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.409 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.409 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.410 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:06:57.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.435 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.436 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:06:57.486 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.486 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.487 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.487 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.488 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.488 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.488 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.489 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.489 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.489 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.490 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.490 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.491 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.491 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.491 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.492 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.492 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.492 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.492 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.493 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.493 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.493 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.494 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.494 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.495 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.495 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.495 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.496 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.496 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.496 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.496 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.497 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.497 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.497 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.498 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.498 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.499 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.499 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.499 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.500 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:06:57.500 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.500 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.500 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.501 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:06:57.501 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.501 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.502 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.502 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:06:57.503 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:06:57.506 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:06:57.585 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:06:57.586 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:06:57.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:06:57.615 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:06:57.615 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:06:57.615 INFO 2458 --- [ocp-schedules-19,1a4c087e6a6b99bb,9c417a4a25a837f7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:06:58.483 WARN 2458 --- [ocp-async-3,1418c6ea7a350ab2,da1b497942eb7780] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:06:58.483 INFO 2458 --- [ocp-async-10,309c36fee1948c74,8db4b9269abcb221] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:06:58.509 INFO 2458 --- [ocp-async-1,0a7a38e0b86b4dd1,83d1b76a3bf8e74c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:06:59.174 INFO 2458 --- [ocp-schedules-9,a8b85149b3b888af,f2d29d6d099553b8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:06:59.575 INFO 2458 --- [ocp-schedules-3,f7e1d4006b269aeb,c22238a25b3a7bd1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:06:59.576 INFO 2458 --- [ocp-schedules-3,f7e1d4006b269aeb,c22238a25b3a7bd1] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:06:59.707 INFO 2458 --- [prometheus-minute-schedule-1,c3403d39e106ae91,c631635d07f71348] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534420, interval=60 2025-10-27 11:06:59.716 INFO 2458 --- [prometheus-minute-schedule-1,c3403d39e106ae91,c631635d07f71348] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:06:59.971 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:06:59.972 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:00.483 WARN 2458 --- [ocp-async-1,00127b18feb415c5,cf43a16c97bb5d1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:00.483 INFO 2458 --- [ocp-async-1,a6ada2425880b752,328aad49b86d3767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:00.555 INFO 2458 --- [ocp-schedules-2,588cc595e90b0d5c,67f6a87f41628e5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:00.642 WARN 2458 --- [scanner-112132392-1,588cc595e90b0d5c,db6bcaac9a60832d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:00.742 INFO 2458 --- [alarm-detect-9,588cc595e90b0d5c,9488c915f6fa1d94] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534390 ~ 1761534410 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534405, value=55.937)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534405, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534405=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:07:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:01.713575+08:00, lastProcessTime=2025-10-27T11:06:51.720382+08:00 2025-10-27 11:07:01.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:01.713575+08:00 2025-10-27 11:07:02.032 INFO 2458 --- [ocp-schedules-26,2bd40dd077a5c16c,9c802de43b35bbe8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:07:02.038 INFO 2458 --- [ocp-schedules-26,2bd40dd077a5c16c,9c802de43b35bbe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:02.073 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:02.073 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:07:02.100 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:02.100 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:02.100 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:02.101 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:02.102 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:02.103 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:07:02.119 INFO 2458 --- [ocp-schedules-26,2bd40dd077a5c16c,9c802de43b35bbe8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:07:02.483 WARN 2458 --- [ocp-async-1,874e510191de686f,992785cfde403668] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:02.483 INFO 2458 --- [ocp-async-1,ed779045edbaba16,b6e0fd6358ad79b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:02.707 INFO 2458 --- [prometheus-minute-schedule-1,33de9d98c5295563,c22b4878d5796a8e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:02.716 INFO 2458 --- [prometheus-minute-schedule-1,33de9d98c5295563,c22b4878d5796a8e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:02.972 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:02.973 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:03.535 INFO 2458 --- [ocp-schedules-30,9d1f14c6e98fb0bd,ade72a8dead13f0b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:03.538 INFO 2458 --- [ocp-schedules-30,9d1f14c6e98fb0bd,ade72a8dead13f0b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:04.483 INFO 2458 --- [ocp-async-7,61859852733c62de,f669bf759b1aa82f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:04.483 WARN 2458 --- [ocp-async-10,f6b6a25f348387bc,75d255846c6a4401] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:05.707 INFO 2458 --- [prometheus-minute-schedule-1,e6726e43708f7ec7,31007941c07ad06b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:05.715 INFO 2458 --- [prometheus-minute-schedule-1,e6726e43708f7ec7,31007941c07ad06b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:05.817 INFO 2458 --- [ocp-schedules-27,d4fa8b6e7e88fca3,cdc14c7feddadb09] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:05.973 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:05.973 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:06.483 INFO 2458 --- [ocp-async-7,ce1cf41787964083,b9246aed19f3087c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:06.483 WARN 2458 --- [ocp-async-2,1b934a3ef6a223d7,2c2633b60a92b128] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:06.665 INFO 2458 --- [ocp-schedules-6,4ed9a046c4036da9,21f0eb3f2906fa13] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:07:06.673 INFO 2458 --- [ocp-schedules-6,4ed9a046c4036da9,21f0eb3f2906fa13] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:06.673 INFO 2458 --- [ocp-schedules-6,4ed9a046c4036da9,21f0eb3f2906fa13] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:08.483 INFO 2458 --- [ocp-async-7,d2e9cbe343ce2a39,0d78eac73e857858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:08.483 WARN 2458 --- [ocp-async-4,5e58477fffe510b7,1afe27fe76c77f5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:08.509 INFO 2458 --- [ocp-async-10,fe7d1000694445c1,a4cd87b122e19b54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:08.707 INFO 2458 --- [prometheus-minute-schedule-1,b9d8c3af04540a0d,82653277470800f5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:08.715 INFO 2458 --- [prometheus-minute-schedule-1,b9d8c3af04540a0d,82653277470800f5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:08.973 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:08.974 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:10.013 INFO 2458 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:07:10.013 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:07:10.051 INFO 2458 --- [Collect obproxy1,136b44e63bdfc1e6,09d6925568b645b1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:07:10.051 INFO 2458 --- [Check obproxy1,e12851503eaba073,e5c1857fe6e5e2fa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:07:10.051 INFO 2458 --- [Collect obproxy1,136b44e63bdfc1e6,09d6925568b645b1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:07:10.051 INFO 2458 --- [Check obproxy1,e12851503eaba073,e5c1857fe6e5e2fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:07:10.087 INFO 2458 --- [Collect obproxy1,136b44e63bdfc1e6,09d6925568b645b1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:07:10.087 INFO 2458 --- [Check obproxy1,e12851503eaba073,e5c1857fe6e5e2fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:07:10.091 INFO 2458 --- [Collect obproxy1,136b44e63bdfc1e6,09d6925568b645b1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:07:10.091 INFO 2458 --- [Check obproxy1,e12851503eaba073,e5c1857fe6e5e2fa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:07:10.091 INFO 2458 --- [Check obproxy1,e12851503eaba073,e5c1857fe6e5e2fa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:07:10.093 INFO 2458 --- [Collect obproxy1,136b44e63bdfc1e6,09d6925568b645b1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:07:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:07:10.501 WARN 2458 --- [ocp-async-10,6f5af6e7e607b295,12c51e11adec81bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:10.501 INFO 2458 --- [ocp-async-6,818dd3fe0d547852,190002a9fd872b8e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:10.555 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534360, nodeCount=47, affectRows=47 2025-10-27 11:07:10.573 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534360, nodeCount=47, affectRows=47 2025-10-27 11:07:10.749 INFO 2458 --- [ocp-schedules-9,439c10f73c7e3c10,d9040893300c2591] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:10.803 WARN 2458 --- [scanner-112133261-1,439c10f73c7e3c10,e510f12cb330cb7f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:10.917 INFO 2458 --- [alarm-detect-11,439c10f73c7e3c10,5463bfab0a770886] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534405 ~ 1761534420 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534420, value=46.191)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534420, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534420=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:07:11.707 INFO 2458 --- [prometheus-minute-schedule-1,c9b90a3fe93d55f8,27ef758e01d6fc11] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:11.715 INFO 2458 --- [prometheus-minute-schedule-1,c9b90a3fe93d55f8,27ef758e01d6fc11] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:11.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:11.717120+08:00, lastProcessTime=2025-10-27T11:07:01.713575+08:00 2025-10-27 11:07:11.945 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:11.717120+08:00 2025-10-27 11:07:11.974 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:11.975 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:12.501 WARN 2458 --- [ocp-async-9,16f9f71bf1ad5042,29ce56b7266d3a4b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:12.501 INFO 2458 --- [ocp-async-5,1602979477d00100,6456ecc2d0e11d0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:13.141 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=1000, affectRows=1000 2025-10-27 11:07:13.145 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=994, validCount=992 2025-10-27 11:07:13.205 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=1000, affectRows=1000 2025-10-27 11:07:13.209 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=999, validCount=995 2025-10-27 11:07:13.223 ERROR 2458 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570101) Duplicate entry '835370602979598336-1761534300' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570101) Duplicate entry '835370602979598336-1761534300' for key 'PRIMARY' 2025-10-27 11:07:13.546 INFO 2458 --- [ocp-schedules-25,ec27514ad1c8c110,7dfab0b993a1bdd2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:13.550 INFO 2458 --- [ocp-schedules-25,ec27514ad1c8c110,7dfab0b993a1bdd2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:13.599 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534360, nodeCount=727, affectRows=727 2025-10-27 11:07:13.839 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:07:13.839 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:07:13.849 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.850 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:07:13.852 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.852 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.853 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.854 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:07:13.863 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.864 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.865 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.865 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:07:13.872 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:07:13.872 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:07:13.873 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.874 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:07:13.876 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.876 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.877 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.878 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:07:13.897 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.897 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.898 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.899 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:07:13.915 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.916 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.917 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.918 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:07:13.933 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.934 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.934 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.935 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:07:13.951 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.951 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.952 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.953 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:07:13.967 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.967 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.968 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.969 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:07:13.985 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.985 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:13.986 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:13.986 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:07:14.000 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.001 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.002 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.002 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:07:14.009 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.009 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.010 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.011 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:07:14.018 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.019 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.019 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.020 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:07:14.026 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.026 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.027 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.027 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:07:14.033 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.034 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.034 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.035 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:07:14.042 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.043 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.043 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.044 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:07:14.051 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.052 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.052 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.053 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:07:14.059 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.060 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.060 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.061 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:07:14.067 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.068 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.069 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.069 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:07:14.075 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.076 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.077 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.077 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:07:14.084 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.084 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.085 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.085 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:07:14.094 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.095 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.096 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.096 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:07:14.106 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.107 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.108 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.109 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:07:14.122 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.122 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.123 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.124 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:07:14.130 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.130 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.131 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.131 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:07:14.138 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.139 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.140 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.140 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:07:14.146 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.147 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.147 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.148 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:07:14.154 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.154 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.155 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.155 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:07:14.161 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.161 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.162 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.163 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:07:14.168 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.169 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.170 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.170 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:07:14.176 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.177 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.177 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.178 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:07:14.183 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.184 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.184 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.185 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:07:14.191 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.191 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.192 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.193 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:07:14.198 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.198 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.199 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.200 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:07:14.205 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.206 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.206 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.207 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:07:14.212 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.213 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.213 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.214 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:07:14.219 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.220 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.221 INFO 2458 --- [ocp-schedules-4,e23a358da94b65ce,1bd66aba433beef3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:07:14.221 INFO 2458 --- [ocp-schedules-4,e23a358da94b65ce,1bd66aba433beef3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:07:14.221 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.221 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:07:14.227 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.228 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.228 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.229 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:07:14.235 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.236 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.236 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.237 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:07:14.242 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.243 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.244 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.244 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:07:14.249 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.250 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.251 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.251 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:07:14.257 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.257 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.258 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.259 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:07:14.265 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.266 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.266 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.267 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:07:14.273 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.273 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.274 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.275 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:07:14.280 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.280 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.281 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.282 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:07:14.283 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:07:14.283 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:07:14.283 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:07:14.283 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:07:14.284 INFO 2458 --- [Collect all tenants major compactions6,e23a358da94b65ce,e5aa155cfb438601] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:07:14.284 INFO 2458 --- [Collect all tenants major compactions5,e23a358da94b65ce,10f7efae6532dd63] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:07:14.286 INFO 2458 --- [ocp-schedules-26,3ac403cfdffa8e25,0d9b3df4dba680f3] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:07:14.286+08:00, clusters=[1, 2] 2025-10-27 11:07:14.288 INFO 2458 --- [ocp-schedules-26,3ac403cfdffa8e25,0d9b3df4dba680f3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:07:14.289 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.289 INFO 2458 --- [ocp-schedules-26,3ac403cfdffa8e25,0d9b3df4dba680f3] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:07:14.291 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.293 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.294 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:07:14.303 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.303 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.304 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.305 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:07:14.310 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.311 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.312 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.313 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:07:14.319 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.320 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.321 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.321 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:07:14.326 INFO 2458 --- [Collect all tenants major compactions5,e23a358da94b65ce,10f7efae6532dd63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.326 INFO 2458 --- [Collect all tenants major compactions6,e23a358da94b65ce,e5aa155cfb438601] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.327 INFO 2458 --- [Collect all tenants major compactions5,e23a358da94b65ce,10f7efae6532dd63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:07:14.327 INFO 2458 --- [Collect all tenants major compactions6,e23a358da94b65ce,e5aa155cfb438601] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:07:14.328 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.329 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.330 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.330 INFO 2458 --- [Collect all tenants major compactions5,e23a358da94b65ce,10f7efae6532dd63] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:07:14.330 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:07:14.330 INFO 2458 --- [Collect all tenants major compactions6,e23a358da94b65ce,e5aa155cfb438601] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:07:14.336 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.337 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.337 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.338 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:07:14.338 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.338 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.338 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.338 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.338 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.340 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:07:14.341 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.341 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.341 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.341 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.341 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.342 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:07:14.343 INFO 2458 --- [Collect all tenants major compactions2,e23a358da94b65ce,f51fa325a552a4df] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:07:14.343 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.344 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.344 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.344 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.344 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:14.345 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.346 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:07:14.346 INFO 2458 --- [Collect all tenants major compactions1,e23a358da94b65ce,e134864c7c862af7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:07:14.346 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:07:14.349 INFO 2458 --- [Collect all tenants major compactions3,e23a358da94b65ce,9f392545a1e5f207] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:07:14.349 INFO 2458 --- [Collect all tenants major compactions4,e23a358da94b65ce,481db7710fe7509f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:07:14.350 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.351 INFO 2458 --- [ocp-schedules-4,e23a358da94b65ce,1bd66aba433beef3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:07:14.351 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.351 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.352 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:07:14.357 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.358 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.358 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.359 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:07:14.365 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.365 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.366 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.366 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:07:14.372 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.372 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.373 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.373 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:07:14.379 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.380 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.380 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.381 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:07:14.388 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.388 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.389 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.390 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:07:14.394 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.394 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.395 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.396 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:07:14.401 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.402 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:07:14.403 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:14.403 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:07:14.407 INFO 2458 --- [ocp-schedules-24,c3c54e6fde0f6543,afea7a36559cd790] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:07:14.501 WARN 2458 --- [ocp-async-7,cf80f43fb5646c68,196ebeae708e586c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:14.501 INFO 2458 --- [ocp-async-5,002816b79c4f5969,2e7fdaa6f220f2d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:14.707 INFO 2458 --- [prometheus-minute-schedule-1,bbbd26d949df2a93,7d42b75cbaf921e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:14.714 INFO 2458 --- [prometheus-minute-schedule-1,bbbd26d949df2a93,7d42b75cbaf921e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:14.734 INFO 2458 --- [ocp-schedules-30,e913619daabdbdf5,3765a0ccba219000] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:07:14.734 INFO 2458 --- [ocp-schedules-30,e913619daabdbdf5,3765a0ccba219000] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:07:14.742 INFO 2458 --- [ocp-schedules-30,e913619daabdbdf5,3765a0ccba219000] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:07:14.742 INFO 2458 --- [ocp-schedules-30,e913619daabdbdf5,3765a0ccba219000] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:07:14.765 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=1000, affectRows=1000 2025-10-27 11:07:14.770 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=1000, validCount=996 2025-10-27 11:07:14.773 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=1000, affectRows=1000 2025-10-27 11:07:14.777 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=992, validCount=988 2025-10-27 11:07:14.785 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=80, affectRows=80 2025-10-27 11:07:14.785 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=78, validCount=78 2025-10-27 11:07:14.975 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:14.977 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:15.825 INFO 2458 --- [ocp-schedules-14,e9e3f9d1f4c69a12,af59ac5bb78fe31e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:16.501 INFO 2458 --- [ocp-async-5,931ee9909fbf0ed4,436a5f1c6bb8a241] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:16.501 WARN 2458 --- [ocp-async-2,24800e0d307b7799,99fcc79cd39d1243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:17.581 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534360, nodeCount=1000, affectRows=1000 2025-10-27 11:07:17.585 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534360, nodeCount=1000, validCount=987 2025-10-27 11:07:17.707 INFO 2458 --- [prometheus-minute-schedule-1,ba1a8ff2765fc107,18d3a60dda1d30b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:17.715 INFO 2458 --- [prometheus-minute-schedule-1,ba1a8ff2765fc107,18d3a60dda1d30b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:17.977 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:17.978 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:18.502 WARN 2458 --- [ocp-async-6,c9a853ecaa6c7817,0c0deca50a1abcba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:18.502 INFO 2458 --- [ocp-async-5,adac9892cc0c834d,6ac869126a696ceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:18.509 INFO 2458 --- [ocp-async-4,d7b48b3b952cbd26,ca106549e1066e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:19.161 INFO 2458 --- [ocp-schedules-22,8c07e65463e511d0,405dca984a8593d0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:07:20.502 WARN 2458 --- [ocp-async-5,932949c6ab1bdddc,b28cb502a6003dc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:20.502 INFO 2458 --- [ocp-async-1,fb2f9f9e98e47ab5,a4c660f37fd9d5aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:20.707 INFO 2458 --- [prometheus-minute-schedule-1,4b25f50f033f4383,2408d991f6e2bca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:20.715 INFO 2458 --- [prometheus-minute-schedule-1,4b25f50f033f4383,2408d991f6e2bca1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:20.924 INFO 2458 --- [ocp-schedules-30,26d67918d8e096f1,756820c49148d08f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:20.966 WARN 2458 --- [scanner-112134361-1,26d67918d8e096f1,a9598206e5302c51] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:20.978 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:20.979 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:21.414 INFO 2458 --- [ocp-schedules-23,37856e7ff1de7d59,54547f809bfcd613] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:21.414 INFO 2458 --- [ocp-schedules-14,e3be164e5da73f26,95b43e306dcc9046] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:21.414 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:07:21.414 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:07:21.415 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:07:21.416 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:07:21.416 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:07:21.426 INFO 2458 --- [ocp-schedules-27,aa44af89c572dc67,a87fdf33bb028dd3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:21.426 INFO 2458 --- [host-check-clockdiff1,aa44af89c572dc67,834da6a9d3453180] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:07:21.427 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:21.427 INFO 2458 --- [host-check-clockdiff2,aa44af89c572dc67,6664706194f8a138] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:07:21.427 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:21.428 INFO 2458 --- [host-check-clockdiff3,aa44af89c572dc67,c29dbfd2bc412f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:07:21.428 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:21.428 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:21.428 INFO 2458 --- [host-check-clockdiff5,aa44af89c572dc67,afd04966fbdaaf4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:07:21.429 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:21.429 INFO 2458 --- [host-check-clockdiff4,aa44af89c572dc67,62484288e5f27dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:07:21.429 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:21.430 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:21.436 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:21.436 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:21.436 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:21.436 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:21.436 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:21.437 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:21.437 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:21.438 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:21.445 INFO 2458 --- [ocp-schedules-1,b7d2375b7eb9966f,7391b144abd8be86] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:07:21.445 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:21.445 INFO 2458 --- [host-check-clockdiff1,aa44af89c572dc67,834da6a9d3453180] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534441 0 0 , error= 2025-10-27 11:07:21.445 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:21.446 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:21.452 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:21.454 INFO 2458 --- [host-check-clockdiff5,aa44af89c572dc67,afd04966fbdaaf4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534441 0 0 , error= 2025-10-27 11:07:21.454 INFO 2458 --- [host-check-clockdiff3,aa44af89c572dc67,c29dbfd2bc412f26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534441 -1 -1 , error= 2025-10-27 11:07:21.454 INFO 2458 --- [host-check-clockdiff4,aa44af89c572dc67,62484288e5f27dec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534441 0 0 , error= 2025-10-27 11:07:21.462 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:21.462 INFO 2458 --- [host-check-clockdiff2,aa44af89c572dc67,6664706194f8a138] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534441 -1 -1 , error= 2025-10-27 11:07:21.512 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:07:21.512 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:07:21.512 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:07:21.512 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:07:21.514 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:07:21.513841743+08:00 2025-10-27 11:07:21.514 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:07:21.514580213+08:00 2025-10-27 11:07:21.514 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:07:21.514839177+08:00 2025-10-27 11:07:21.515 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:07:21.515+08:00 2025-10-27 11:07:21.515 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:07:21.514+08:00 2025-10-27 11:07:21.515 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:07:21.514+08:00 2025-10-27 11:07:21.516 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:07:21.517 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:07:21.517 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:07:21.517 INFO 2458 --- [host-check-agent-status2,e3be164e5da73f26,2017b2cc9f6db358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:07:21.518 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:07:21.518 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:07:21.517625056+08:00 2025-10-27 11:07:21.519 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:07:21.518+08:00 2025-10-27 11:07:21.521 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:07:21.521 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:07:21.522 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:07:21.522 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:07:21.522140951+08:00 2025-10-27 11:07:21.523 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:07:21.522+08:00 2025-10-27 11:07:21.523 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:07:21.514+08:00 and status AVAILABLE 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status4,e3be164e5da73f26,37398fbfb9303a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:07:21.514+08:00 and status AVAILABLE 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:07:21.524 INFO 2458 --- [host-check-agent-status3,e3be164e5da73f26,6ea2c18db1bde524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:07:21.526 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:07:21.515+08:00 and status AVAILABLE 2025-10-27 11:07:21.527 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:07:21.518+08:00 and status AVAILABLE 2025-10-27 11:07:21.527 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:07:21.527 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:07:21.527 INFO 2458 --- [host-check-agent-status5,e3be164e5da73f26,545fd92f55b078d9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:07:21.528 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:07:21.528 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:07:21.528 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:07:21.528 INFO 2458 --- [host-check-agent-status1,e3be164e5da73f26,864cfd2cc9e71701] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:07:21.530 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:07:21.522+08:00 and status AVAILABLE 2025-10-27 11:07:21.530 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:07:21.532 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:07:21.532 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:07:21.536 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:07:21.545 INFO 2458 --- [ocp-schedules-5,aca039e3dc0353a4,1eb0e49cfdfed49a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:07:21.548 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:21.549 INFO 2458 --- [ocp-schedules-17,16f2cb3b55155b43,d321d1f2b44ce616] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:07:21.550 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:21.550 INFO 2458 --- [ocp-schedules-5,aca039e3dc0353a4,1eb0e49cfdfed49a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:21.551 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:21.551 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:21.552 INFO 2458 --- [ocp-schedules-17,16f2cb3b55155b43,d321d1f2b44ce616] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:21.553 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:21.553 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:21.553 INFO 2458 --- [ocp-schedules-17,16f2cb3b55155b43,d321d1f2b44ce616] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:07:21.554 INFO 2458 --- [ocp-schedules-19,64a52836907f858b,368ea48a8f4256bd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:07:21.554 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:21.555 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:21.555 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:21.558 INFO 2458 --- [ocp-schedules-19,64a52836907f858b,368ea48a8f4256bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:21.558 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:21.590 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.591 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:21.592 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:07:21.613 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.613 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.613 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.614 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.614 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.614 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:21.615 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:21.615 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:21.616 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:21.619 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:07:21.620 INFO 2458 --- [ocp-schedules-5,aca039e3dc0353a4,1eb0e49cfdfed49a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:07:21.621 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:21.626 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.626 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:21.636 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:07:21.637 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:07:21.638 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:07:21.641 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:07:21.646 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:07:21.666 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:07:21.666 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:07:21.670 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:07:21.670 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.670 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:07:21.670 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.671 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.671 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.671 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:21.676 INFO 2458 --- [ocp-schedules-13,457040e6b0c41118,77bf9166c12b2769] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:07:21.679 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:07:21.683 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:21.683 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.687 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:07:21.705 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:21.705 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:21.706 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:21.707 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:21.715794+08:00, lastProcessTime=2025-10-27T11:07:11.717120+08:00 2025-10-27 11:07:21.720 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:07:21.723 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:21.724 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:07:21.727 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:07:21.730 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:07:21.750 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:21.751 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.751 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:21.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.756 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:07:21.756 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:21.757 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:07:21.758 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:07:21.760 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:07:21.763 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:07:21.765 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.769 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.770 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:07:21.778 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:21.778 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:21.792 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:21.794 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:07:21.796 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:21.796 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:21.797 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:21.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:21.807 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:07:21.814 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:07:21.818 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:07:21.818 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:07:21.825 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:21.840 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.841 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:21.842 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:07:21.845 INFO 2458 --- [host-check-host-status4,37856e7ff1de7d59,d076b2597a79ae8d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:07:21.849 INFO 2458 --- [host-check-host-status2,37856e7ff1de7d59,cc686c6e0e1eeec9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:07:21.851 INFO 2458 --- [host-check-host-status3,37856e7ff1de7d59,e6aa21d0d6f8f35c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:07:21.852 INFO 2458 --- [host-check-host-status5,37856e7ff1de7d59,481f92e685d1ac29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:07:21.854 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:07:21.856 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:07:21.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:21.867 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:21.872 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:21.872 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:21.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:07:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:21.884 INFO 2458 --- [host-check-host-status1,37856e7ff1de7d59,ef92f7438803633a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:07:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:21.921 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:07:21.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:21.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:21.965 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:07:21.966 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.967 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:07:21.968 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:21.969 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:07:21.977 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:21.715794+08:00 2025-10-27 11:07:21.986 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:07:21.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:21.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:22.008 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:22.009 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:22.009 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:07:22.043 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:07:22.043 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:07:22.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:22.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:22.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:22.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:22.130 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:22.170 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:22.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:07:22.187 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:07:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:22.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:22.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:07:22.302 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:07:22.302 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.302 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.302 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:22.303 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:07:22.305 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.305 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.305 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:22.306 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:07:22.340 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:07:22.377 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:22.377 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.377 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.378 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:22.378 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:22.378 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:22.379 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:07:22.429 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:07:22.429 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:07:22.429 INFO 2458 --- [ocp-schedules-19,64a52836907f858b,368ea48a8f4256bd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:07:22.502 INFO 2458 --- [ocp-async-8,2eabc88edd4443f8,97c34996ac0c0275] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:22.502 WARN 2458 --- [ocp-async-10,15c1b2a8d30428bd,3271fcd90e2bf236] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:23.558 INFO 2458 --- [ocp-schedules-2,f4ef90dfed1d4a18,c30142123b2472ba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:23.562 INFO 2458 --- [ocp-schedules-2,f4ef90dfed1d4a18,c30142123b2472ba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:23.707 INFO 2458 --- [prometheus-minute-schedule-1,f414fdc87d147b3e,6c4288a5ebca972e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:23.715 INFO 2458 --- [prometheus-minute-schedule-1,f414fdc87d147b3e,6c4288a5ebca972e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:23.979 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:23.980 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:24.502 INFO 2458 --- [ocp-async-6,558fd5b64ca21685,2f3ee3cc80929f05] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:24.502 WARN 2458 --- [ocp-async-7,b9074172c5f123cf,bd8b6333b826db5f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:25.831 INFO 2458 --- [ocp-schedules-6,3c98533dfa548ff1,31c49042fe9fa255] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:26.502 WARN 2458 --- [ocp-async-9,629ecba1ed9a65a3,174fe6ba6996ff35] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:26.502 INFO 2458 --- [ocp-async-9,0f428a6caddd2c4a,d8e387522db59326] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:26.675 INFO 2458 --- [ocp-schedules-4,72405c8110568086,7067ff9a7c6495d5] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:07:26.683 INFO 2458 --- [ocp-schedules-4,72405c8110568086,7067ff9a7c6495d5] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:26.683 INFO 2458 --- [ocp-schedules-4,72405c8110568086,7067ff9a7c6495d5] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:26.707 INFO 2458 --- [prometheus-minute-schedule-1,809c2dbfac571a85,59eb859863310451] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:26.715 INFO 2458 --- [prometheus-minute-schedule-1,809c2dbfac571a85,59eb859863310451] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:26.980 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:26.981 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:28.502 WARN 2458 --- [ocp-async-10,f7c869775f913aae,36319133cdd95be3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:28.502 INFO 2458 --- [ocp-async-1,0894a3193b692cff,849a84b73977a487] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:28.509 INFO 2458 --- [ocp-async-8,da2b1c28824c4410,0ea9721160753fcf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:28.842 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:07:28.842 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:07:28.844 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:07:28.844 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:07:28.846 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:07:28.846 INFO 2458 --- [ocp-schedules-19,abf8a57ad3b8ff6f,54f161ec3b36bf67] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:07:29.307 INFO 2458 --- [ocp-schedules-27,15dd3746e6a822ad,e4f24dadb9eeb517] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:07:29.312 INFO 2458 --- [ocp-schedules-27,15dd3746e6a822ad,e4f24dadb9eeb517] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:07:29.312 INFO 2458 --- [ocp-schedules-27,15dd3746e6a822ad,e4f24dadb9eeb517] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:07:29.707 INFO 2458 --- [prometheus-minute-schedule-1,b53280de1856c4f2,f89423e992e7a172] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:29.719 INFO 2458 --- [prometheus-minute-schedule-1,b53280de1856c4f2,f89423e992e7a172] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:29.981 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:29.982 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:30.502 INFO 2458 --- [ocp-async-7,83380af5241cb8a5,a3d81a8ee93f9669] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:30.503 WARN 2458 --- [ocp-async-3,c19975f1ee61fa1e,3cda563753f2184e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:31.094 INFO 2458 --- [ocp-schedules-30,a4c522618efefe84,93d5a4f6fe59b17c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:31.200 WARN 2458 --- [scanner-112134593-1,a4c522618efefe84,1a16ad8994551a7c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:31.460 INFO 2458 --- [alarm-detect-5,a4c522618efefe84,6777eab147522b34] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534420 ~ 1761534441 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534435, value=51.533)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534435, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534435=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:07:31.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:31.713079+08:00, lastProcessTime=2025-10-27T11:07:21.715794+08:00 2025-10-27 11:07:31.924 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:31.713079+08:00 2025-10-27 11:07:32.503 INFO 2458 --- [ocp-async-4,da3aaaf6a11ed724,bae14ac92e79c0c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:32.503 WARN 2458 --- [ocp-async-9,fbeaec419af1ceb1,a4034931f9516485] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:32.707 INFO 2458 --- [prometheus-minute-schedule-1,6be2551c116c60e3,063fd909ecbe02b1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:32.715 INFO 2458 --- [prometheus-minute-schedule-1,6be2551c116c60e3,063fd909ecbe02b1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:32.982 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:32.983 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:33.570 INFO 2458 --- [ocp-schedules-25,42f10d9e2211d8de,b3574f264d677c40] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:33.573 INFO 2458 --- [ocp-schedules-25,42f10d9e2211d8de,b3574f264d677c40] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:33.736 INFO 2458 --- [ocp-schedules-1,22ef3d54f6408dbc,4baf9706b9511e01] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:07:33.736 INFO 2458 --- [ocp-schedules-1,22ef3d54f6408dbc,4baf9706b9511e01] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:07:34.503 WARN 2458 --- [ocp-async-2,6a6c681af9b96c94,73acb6ab5ae2f450] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:34.503 INFO 2458 --- [ocp-async-1,4912c1e3ec72938e,2f7979a798686460] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:34.913 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:07:34.913 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:07:34.986 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:34.986 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:35.031 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:35.031 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:35.031 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:35.032 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:07:35.045 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:35.045 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:35.045 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:35.046 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:07:35.115 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:35.116 INFO 2458 --- [ocp-schedules-30,4533b537d3b0fd82,ed0e8eb67181f4fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:07:35.375 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,17fd8f7a8bacc4c1,f51bc85e6cb9eb7e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=17fd8f7a8bacc4c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:07:35.375 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,17fd8f7a8bacc4c1,f51bc85e6cb9eb7e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=17fd8f7a8bacc4c1, duration=1 ms] 2025-10-27 11:07:35.707 INFO 2458 --- [prometheus-minute-schedule-1,815360bae5db06fe,b9559e514b6000c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:35.737 INFO 2458 --- [prometheus-minute-schedule-1,815360bae5db06fe,b9559e514b6000c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:35.838 INFO 2458 --- [ocp-schedules-12,6c9913a55deb9ad1,5f5c58ce828f2601] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:35.983 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:35.984 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:36.503 INFO 2458 --- [ocp-async-3,c41b298e26c914ca,5a6dd8587bac8558] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:36.503 WARN 2458 --- [ocp-async-6,be7b5289c1abf051,272649e37e8a6f33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:36.966 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:36.966 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:07:36.966 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:07:36.966 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:07:36.966 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:07:36.966 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:07:36.977 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:36.978 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:36.980 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:36.980 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:36.981 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:36.981 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:36.982 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:36.983 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:36.983 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:36.985 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:37.034 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:07:37.034 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:07:37.036 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:07:37.037 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:07:37.040 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:07:37.041 INFO 2458 --- [ocp-schedules-10,3dcbc4b4e3e958cd,f5ec9be5bc825665] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:07:37.708 INFO 2458 --- [prometheus-cache-monitor1,bf3a5f6e9bea6c64,87cec0fc24cc608b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20304,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:07:37.708 INFO 2458 --- [prometheus-cache-monitor1,bf3a5f6e9bea6c64,87cec0fc24cc608b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:07:38.503 WARN 2458 --- [ocp-async-7,2f47e604ea042870,719db9153e253683] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:38.503 INFO 2458 --- [ocp-async-5,6dd0cab511c94ea9,86d51bd172722b3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:38.509 INFO 2458 --- [ocp-async-4,83aec53971f9e1a6,2de91b9e64982fab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:38.584 INFO 2458 --- [ocp-schedules-11,929e8888f7712eda,b93545992c47e159] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:07:38.592 INFO 2458 --- [ocp-schedules-11,929e8888f7712eda,b93545992c47e159] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:38.592 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:07:38.592 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:07:38.592 INFO 2458 --- [ocp-schedules-11,929e8888f7712eda,b93545992c47e159] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:07:38.707 INFO 2458 --- [prometheus-minute-schedule-1,171e30076c46f00d,c72857450013b16b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:38.713 INFO 2458 --- [prometheus-minute-schedule-1,171e30076c46f00d,c72857450013b16b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:38.984 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:38.985 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:40.013 INFO 2458 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:07:40.013 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:07:40.048 INFO 2458 --- [Check obproxy1,16cef262d61e6900,c6824f65f3dbd4ee] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:07:40.048 INFO 2458 --- [Check obproxy1,16cef262d61e6900,c6824f65f3dbd4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:07:40.059 INFO 2458 --- [Collect obproxy1,0dfa390f2a908ab1,f7c18b90cd1ee4c4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:07:40.059 INFO 2458 --- [Collect obproxy1,0dfa390f2a908ab1,f7c18b90cd1ee4c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:07:40.080 INFO 2458 --- [Check obproxy1,16cef262d61e6900,c6824f65f3dbd4ee] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:07:40.083 INFO 2458 --- [Check obproxy1,16cef262d61e6900,c6824f65f3dbd4ee] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:07:40.083 INFO 2458 --- [Check obproxy1,16cef262d61e6900,c6824f65f3dbd4ee] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:07:40.087 INFO 2458 --- [Collect obproxy1,0dfa390f2a908ab1,f7c18b90cd1ee4c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:07:40.090 INFO 2458 --- [Collect obproxy1,0dfa390f2a908ab1,f7c18b90cd1ee4c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:07:40.093 INFO 2458 --- [Collect obproxy1,0dfa390f2a908ab1,f7c18b90cd1ee4c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:07:40.376 INFO 2458 --- [ocp-schedules-11,7207b620d84602ee,70072c19e198b2dc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:07:40.377 INFO 2458 --- [ocp-schedules-11,7207b620d84602ee,70072c19e198b2dc] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:07:40.401 INFO 2458 --- [ocp-schedules-11,7207b620d84602ee,70072c19e198b2dc] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:07:40.503 WARN 2458 --- [ocp-async-10,d1687fdbf3dc5f03,f5429e3422149a37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:40.503 INFO 2458 --- [ocp-async-1,33148b951c3629f9,10e32777b96f1ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:41.477 INFO 2458 --- [ocp-schedules-7,cc93782aa6ba8309,adf936f97d24b1d0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:41.505 WARN 2458 --- [scanner-112134794-1,cc93782aa6ba8309,01dc8d78518ce1f7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:41.677 INFO 2458 --- [alarm-detect-13,cc93782aa6ba8309,c32e3ca3ef583204] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534435 ~ 1761534451 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534450, value=69.431)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534450, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534450=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:07:41.707 INFO 2458 --- [prometheus-minute-schedule-1,1afe3569754a74c1,534cfbb5cd7c94fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:41.713 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:41.712033+08:00, lastProcessTime=2025-10-27T11:07:31.713079+08:00 2025-10-27 11:07:41.715 INFO 2458 --- [prometheus-minute-schedule-1,1afe3569754a74c1,534cfbb5cd7c94fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:41.918 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:41.712033+08:00 2025-10-27 11:07:41.985 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:41.985 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:42.503 WARN 2458 --- [ocp-async-8,4a3f6b2b5d2e1c03,43892c404503716a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:42.503 INFO 2458 --- [ocp-async-3,ee2e0c021d5a7451,3503f859f08c4dc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:43.581 INFO 2458 --- [ocp-schedules-19,4388166137a117ee,1827ed8b1a382805] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:43.584 INFO 2458 --- [ocp-schedules-19,4388166137a117ee,1827ed8b1a382805] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:43.983 INFO 2458 --- [ocp-schedules-1,fe7e570d3050432e,46de5a2bf6c81cda] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:07:44.503 WARN 2458 --- [ocp-async-5,a6a989934620e419,6573eb9ca25695c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:44.503 INFO 2458 --- [ocp-async-9,408c40e2fb776d56,7a8b25106a73d748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:44.707 INFO 2458 --- [prometheus-minute-schedule-1,14b390ce222c1814,b752686745d4170b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:44.716 INFO 2458 --- [prometheus-minute-schedule-1,14b390ce222c1814,b752686745d4170b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:44.915 INFO 2458 --- [ocp-schedules-16,7e56434d2984fff8,a5fd9f5fa2d55aa4] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:07:44.915 INFO 2458 --- [ocp-schedules-16,7e56434d2984fff8,a5fd9f5fa2d55aa4] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:07:44.923 INFO 2458 --- [ocp-schedules-16,7e56434d2984fff8,a5fd9f5fa2d55aa4] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:07:44.923 INFO 2458 --- [ocp-schedules-16,7e56434d2984fff8,a5fd9f5fa2d55aa4] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:07:44.986 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:44.986 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:45.102 INFO 2458 --- [prometheus-second-schedule-1,e833647d1bde48c7,650a2973ad6d7d10] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:07:45.845 INFO 2458 --- [ocp-schedules-13,d0797c0eb49692e1,c5cbd9e5a0f06b7f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:46.410 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:07:46.446 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:46.450 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:46.504 INFO 2458 --- [ocp-async-10,cb1c4e45335397d6,e35ed60b1b6ac4fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:46.504 WARN 2458 --- [ocp-async-2,f7812661589e4286,3a290bb6ae3d115e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:46.505 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:46.508 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:46.558 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:46.560 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:46.616 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:46.619 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:46.671 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:46.674 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:46.686 INFO 2458 --- [ocp-schedules-17,c30e43f6cae270ae,64a7c811ae8f0867] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:07:46.694 INFO 2458 --- [ocp-schedules-17,c30e43f6cae270ae,64a7c811ae8f0867] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:46.694 INFO 2458 --- [ocp-schedules-17,c30e43f6cae270ae,64a7c811ae8f0867] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:07:46.734 INFO 2458 --- [ocp-schedules-14,02db03e398d95ad9,dfd51a0a96e36618] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:07:47.707 INFO 2458 --- [prometheus-minute-schedule-1,cfb1f5f2e5460b89,d243d404cd656ea6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:47.715 INFO 2458 --- [prometheus-minute-schedule-1,cfb1f5f2e5460b89,d243d404cd656ea6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:47.738 INFO 2458 --- [ocp-schedules-23,e2ea359726b5b90d,2716370d905391dc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:07:47.738+08:00, clusters=[1, 2] 2025-10-27 11:07:47.752 INFO 2458 --- [ocp-schedules-23,e2ea359726b5b90d,2716370d905391dc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:07:47.755 INFO 2458 --- [ocp-schedules-23,e2ea359726b5b90d,2716370d905391dc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:07:47.986 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:47.987 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:48.504 INFO 2458 --- [ocp-async-8,e8f545f3d7385f5c,13143fff864e33bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:48.504 WARN 2458 --- [ocp-async-6,78599da570104b05,1321f57ebf0d5283] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:48.509 INFO 2458 --- [ocp-async-3,96235c2d0274e3f3,aadb396d5d8ec011] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:50.504 INFO 2458 --- [ocp-async-7,1017bb3e2239e38b,b825a042ccdc9e20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:50.504 WARN 2458 --- [ocp-async-9,983af05d69755aa4,68f54953c3120e6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:50.707 INFO 2458 --- [prometheus-minute-schedule-1,f41333bcc7ef1e66,6ba1a991ccfc4daa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:50.714 INFO 2458 --- [prometheus-minute-schedule-1,f41333bcc7ef1e66,6ba1a991ccfc4daa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:50.987 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:50.988 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:51.413 INFO 2458 --- [ocp-schedules-27,4553b044508e3202,84eb9379431f3abb] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:51.413 INFO 2458 --- [ocp-schedules-30,af4497f25bde7628,27d14ade76a270aa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:51.413 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:07:51.413 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:07:51.414 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:07:51.414 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:07:51.416 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:07:51.419 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:07:51.419 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:07:51.421 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:07:51.422 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:07:51.423 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:07:51.431 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:51.431 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:51.432 INFO 2458 --- [ocp-schedules-13,fcc50444386d79c5,59901196b83b5453] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:07:51.432 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:51.432 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:51.433 INFO 2458 --- [host-check-clockdiff2,fcc50444386d79c5,834af84fff01d0c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:07:51.433 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:51.433 INFO 2458 --- [host-check-clockdiff3,fcc50444386d79c5,d4871d102367f25f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:07:51.433 INFO 2458 --- [host-check-clockdiff4,fcc50444386d79c5,5ae97afcb46d9b37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:07:51.434 INFO 2458 --- [host-check-clockdiff1,fcc50444386d79c5,93ed6baf999f075b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:07:51.436 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:51.437 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:51.437 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:51.437 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:51.438 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:51.438 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:51.439 INFO 2458 --- [host-check-clockdiff5,fcc50444386d79c5,a19d82ab9562797b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:07:51.439 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:51.441 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:51.452 INFO 2458 --- [ocp-schedules-17,04e637eb892f0ed1,f1d1d49ee8c07df4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:07:51.453 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:51.454 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:51.455 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:51.456 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:51.456 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:51.457 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:51.466 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:51.470 INFO 2458 --- [host-check-clockdiff2,fcc50444386d79c5,834af84fff01d0c0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534471 -1 -1 , error= 2025-10-27 11:07:51.476 INFO 2458 --- [host-check-clockdiff4,fcc50444386d79c5,5ae97afcb46d9b37] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534471 -1 -1 , error= 2025-10-27 11:07:51.476 INFO 2458 --- [host-check-clockdiff1,fcc50444386d79c5,93ed6baf999f075b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534471 0 0 , error= 2025-10-27 11:07:51.478 INFO 2458 --- [host-check-clockdiff3,fcc50444386d79c5,d4871d102367f25f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534471 -1 -1 , error= 2025-10-27 11:07:51.519 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:07:51.523 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:07:51.523 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:07:51.523 INFO 2458 --- [host-check-agent-status1,af4497f25bde7628,25dfc8928130474e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:07:51.524 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:07:51.524 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:07:51.525 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:07:51.524838096+08:00 2025-10-27 11:07:51.525 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:07:51.525+08:00 2025-10-27 11:07:51.528 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:07:51.528 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:07:51.528 INFO 2458 --- [host-check-agent-status2,af4497f25bde7628,d8a2c948c6e38bd6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:07:51.529 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:07:51.531 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:07:51.530638880+08:00 2025-10-27 11:07:51.532 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:07:51.532 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:07:51.532+08:00 2025-10-27 11:07:51.534 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:07:51.534 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:07:51.525+08:00 and status AVAILABLE 2025-10-27 11:07:51.535 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:07:51.535708720+08:00 2025-10-27 11:07:51.536 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:07:51.535+08:00 2025-10-27 11:07:51.536 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:07:51.536 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:07:51.536 INFO 2458 --- [host-check-agent-status5,af4497f25bde7628,7fc7e58306d84d9e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:07:51.538 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:07:51.540 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:07:51.540 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:07:51.539748294+08:00 2025-10-27 11:07:51.541 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:07:51.532+08:00 and status AVAILABLE 2025-10-27 11:07:51.541 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:07:51.540+08:00 2025-10-27 11:07:51.542 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:07:51.543 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:07:51.543 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:07:51.535+08:00 and status AVAILABLE 2025-10-27 11:07:51.544 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:07:51.544 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:07:51.544 INFO 2458 --- [host-check-agent-status3,af4497f25bde7628,988ab0cf95dd8de1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:07:51.544 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:07:51.543422501+08:00 2025-10-27 11:07:51.544 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:07:51.544+08:00 2025-10-27 11:07:51.545 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:07:51.545 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:07:51.545 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:07:51.545 INFO 2458 --- [host-check-agent-status4,af4497f25bde7628,8b8e4706d4154449] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:07:51.545 INFO 2458 --- [ocp-schedules-9,216926170d274834,ecd3f2d78ac33e01] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:07:51.548 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:07:51.540+08:00 and status AVAILABLE 2025-10-27 11:07:51.549 INFO 2458 --- [ocp-schedules-10,cf0cf5da6b2dfdad,575d78513b123252] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:07:51.549 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:07:51.550 INFO 2458 --- [ocp-schedules-9,216926170d274834,ecd3f2d78ac33e01] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:51.552 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:07:51.553 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:07:51.544+08:00 and status AVAILABLE 2025-10-27 11:07:51.553 INFO 2458 --- [ocp-schedules-10,cf0cf5da6b2dfdad,575d78513b123252] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:51.553 INFO 2458 --- [ocp-schedules-10,cf0cf5da6b2dfdad,575d78513b123252] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:07:51.554 INFO 2458 --- [ocp-schedules-7,aca505108ddf639b,ecd27d1068480ab4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:07:51.557 INFO 2458 --- [ocp-schedules-7,aca505108ddf639b,ecd27d1068480ab4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:51.561 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:07:51.565 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:07:51.565 INFO 2458 --- [ocp-schedules-16,1edcfa7113f285fb,aabd10f79d221fb6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:07:51.568 INFO 2458 --- [ocp-schedules-29,40eab6fb205aceae,9c5ad1a3fb8b76d9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:07:51.568 INFO 2458 --- [ocp-schedules-4,6777b84dc0096e77,8c75f88ac5f7dc63] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:07:51.569 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:07:51.569 INFO 2458 --- [ocp-schedules-20,79fe297d42a8ffec,039506317e7d9e26] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:07:51.570 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:51.573 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:51.573 INFO 2458 --- [ocp-schedules-29,40eab6fb205aceae,9c5ad1a3fb8b76d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:51.574 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:51.577 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:07:51.578 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:51.580 INFO 2458 --- [ocp-schedules-20,79fe297d42a8ffec,039506317e7d9e26] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:07:51.582 INFO 2458 --- [ocp-schedules-4,6777b84dc0096e77,8c75f88ac5f7dc63] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:07:51.594 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:07:51.598 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:07:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:51.603 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:07:51.607 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:07:51.614 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:07:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.630 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.630 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:51.631 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:51.631 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:07:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:07:51.636 INFO 2458 --- [ocp-schedules-9,216926170d274834,ecd3f2d78ac33e01] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:07:51.643 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,b941f3f93e47fca2,634c3eb30c462a13] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=b941f3f93e47fca2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:07:51.643 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,b941f3f93e47fca2,634c3eb30c462a13] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=b941f3f93e47fca2, duration=1 ms] 2025-10-27 11:07:51.644 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.644 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:51.645 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:07:51.648 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:51.648 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:51.653 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.656 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.656 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.656 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:07:51.657 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.657 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.657 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.658 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:07:51.665 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:07:51.665 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:07:51.682 INFO 2458 --- [ocp-schedules-12,0184c4f79a7bde4a,5e7dd0b936c1851c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:07:51.685 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:07:51.687 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:51.717 WARN 2458 --- [scanner-112135074-1,0184c4f79a7bde4a,4c2277398deb45fa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:07:51.721 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:51.721 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:07:51.721 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.722 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:07:51.724 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:07:51.747 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,580c3c0420a4041f,21672bdfdebfd689] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=580c3c0420a4041f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:07:51.747 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,580c3c0420a4041f,21672bdfdebfd689] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=580c3c0420a4041f, duration=1 ms] 2025-10-27 11:07:51.757 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:07:51.757 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.757 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.757 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.758 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:07:51.777 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:07:51.775886+08:00, lastProcessTime=2025-10-27T11:07:41.712033+08:00 2025-10-27 11:07:51.783 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:07:51.786 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.786 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.786 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.787 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.787 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.788 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:51.789 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:07:51.792 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.792 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.792 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.795 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:07:51.796 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.796 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.797 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.800 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:07:51.801 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:07:51.803 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:07:51.804 INFO 2458 --- [ocp-schedules-29,40eab6fb205aceae,9c5ad1a3fb8b76d9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:07:51.806 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:07:51.819 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.819 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:51.820 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:51.836 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=n 2025-10-27 11:07:51.837 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.839 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:07:51.840 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:51.842 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:07:51.842 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.843 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:51.844 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.846 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.848 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.849 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.850 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:07:51.852 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:07:51.852 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:51.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:51.864 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:51.864 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.869 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:07:51.878 INFO 2458 --- [ocp-schedules-19,77e63f50f3f03723,3e862ba3505c67a8] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:07:51.878 INFO 2458 --- [ocp-schedules-19,77e63f50f3f03723,3e862ba3505c67a8] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:07:51.881 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.939 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:07:51.939 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.942 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:51.942 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:51.942 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:07:51.946 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:07:51.950 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.957 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:07:51.961 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.965 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.966 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.967 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:07:51.967 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:51.967 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.967 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:07:51.968 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.969 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.970 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:07:51.971 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:07:51.976 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.978 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:07:51.981 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:51.985 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.985 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:07:51.986 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:51.987 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:51.990 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:51.995 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.006 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:07:52.006 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:52.007 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:52.009 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:07:52.018 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:07:52.020 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.021 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.022 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:07:52.023 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.024 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.025 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:52.026 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:07:52.026 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.027 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.029 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.030 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.032 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.033 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.037 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:07:52.038 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:07:52.048 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:01:51.61, end=2025-10-27T11:06:51.812} 2025-10-27 11:07:52.048 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:07:52.049 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:07:52.050 INFO 2458 --- [host-check-host-status5,4553b044508e3202,bbe061f27b5cf1a6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:07:52.052 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:07:52.052 INFO 2458 --- [host-check-host-status2,4553b044508e3202,19d24c39a518e918] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:07:52.052 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:07:52.059 INFO 2458 --- [host-check-host-status4,4553b044508e3202,fee8d111e0db89b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:07:52.069 INFO 2458 --- [host-check-host-status1,4553b044508e3202,7e115db3efb24368] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:07:52.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.074 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.075 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:07:52.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.076 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.077 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:07:52.079 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:07:52.080 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:52.105 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:07:52.108 INFO 2458 --- [host-check-host-status3,4553b044508e3202,b6edbac60349cc2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:07:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:52.121 INFO 2458 --- [pool-overview-container-7,786aaf3b9b732ee4,737479cc400c15ee] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.148 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:07:52.148 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.149 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:07:52.150 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:07:51.775886+08:00 2025-10-27 11:07:52.151 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.152 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:07:52.156 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.156 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:52.177 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:07:52.204 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.204 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:52.205 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:07:52.231 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:07:52.231 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:07:52.239 INFO 2458 --- [pool-overview-container-8,80c7f194e938a5ae,a80c9aeaecd110e5] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:07:52.278 INFO 2458 --- [pool-overview-container-5,5b854c62dbe4add4,fd1f73a0be2bfb59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.280 INFO 2458 --- [pool-overview-container-5,5b854c62dbe4add4,fd1f73a0be2bfb59] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:07:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:07:52.305 INFO 2458 --- [pool-12-thread-3,80c7f194e938a5ae,ad6b242a24034f84] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:07:52.307 INFO 2458 --- [pool-12-thread-4,80c7f194e938a5ae,3649df6bb650d629] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:07:52.309 INFO 2458 --- [pool-12-thread-1,80c7f194e938a5ae,bf2c1759c64770d8] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:07:52.311 INFO 2458 --- [ForkJoinPool-7-worker-6,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.314 INFO 2458 --- [ForkJoinPool-7-worker-15,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.315 INFO 2458 --- [pool-12-thread-2,80c7f194e938a5ae,a9c397a6b460fa80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:07:52.315 INFO 2458 --- [pool-12-thread-2,80c7f194e938a5ae,a9c397a6b460fa80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:07:52.317 INFO 2458 --- [ForkJoinPool-7-worker-1,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.318 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:07:52.318 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:07:52.319 INFO 2458 --- [ForkJoinPool-7-worker-8,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:07:52.321 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:07:52.321 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:07:52.326 INFO 2458 --- [pool-12-thread-4,80c7f194e938a5ae,3649df6bb650d629] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:07:52.326 INFO 2458 --- [pool-12-thread-4,80c7f194e938a5ae,3649df6bb650d629] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:07:52.341 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,4d4095d153089bd8,e25e1399f2c27986] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=4d4095d153089bd8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:07:52.342 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,4d4095d153089bd8,e25e1399f2c27986] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=4d4095d153089bd8, duration=1 ms] 2025-10-27 11:07:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:07:52.346 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.348 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:07:52.357 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.358 INFO 2458 --- [pool-12-thread-2,80c7f194e938a5ae,a9c397a6b460fa80] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:07:52.358 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:07:52.359 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.360 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:07:52.365 INFO 2458 --- [pool-12-thread-2,80c7f194e938a5ae,a9c397a6b460fa80] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:07:52.368 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.368 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.369 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.370 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.370 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.373 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:07:52.373 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:07:52.378 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:07:52.379 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.380 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.380 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.390 INFO 2458 --- [host-check-clockdiff5,fcc50444386d79c5,a19d82ab9562797b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534472 1 1 , error= 2025-10-27 11:07:52.380 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.390 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:07:52.391 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.391 INFO 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:07:52.392 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:07:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.400 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.400 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.400 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.400 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.401 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:07:52.407 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:07:52.412 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.413 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:07:52.414 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.414 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:07:52.414 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:07:52.417 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:07:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:07:52.434 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.434 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:07:52.436 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.440 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.442 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:07:52.442 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.442 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.442 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.443 INFO 2458 --- [ForkJoinPool-7-worker-15,80c7f194e938a5ae,00313bb69403dd87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:07:52.445 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:07:52.455 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.456 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:07:52, 2025-10-27 11:07:52] 2025-10-27 11:07:52.458 WARN 2458 --- [ForkJoinPool-7-worker-1,80c7f194e938a5ae,76c13e25e472cdb7] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:07:52.463 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:07:52.464 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.465 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:07:52.466 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.467 INFO 2458 --- [ForkJoinPool-7-worker-6,80c7f194e938a5ae,3ef5e1154d34cfab] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:07:52.482 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:07:52.483 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.484 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:07:52, 2025-10-27 11:07:52] 2025-10-27 11:07:52.486 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:07:52.486 INFO 2458 --- [ForkJoinPool-7-worker-8,80c7f194e938a5ae,844d434b2d08996d] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:07:52.486 INFO 2458 --- [pool-12-thread-1,80c7f194e938a5ae,bf2c1759c64770d8] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:07:52.493 INFO 2458 --- [pool-12-thread-3,80c7f194e938a5ae,ad6b242a24034f84] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:07:52.493 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.493 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.494 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.494 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.494 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.498 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:07:52.498 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:07:52.504 INFO 2458 --- [ocp-async-2,c6702a08bae26225,98a3aeaddad52eeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:52.504 WARN 2458 --- [ocp-async-6,997018a1059fbc87,2556268203608f96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:52.552 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:07:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.553 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:07:52.555 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.555 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.555 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.556 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:07:52.605 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:07:52.645 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.645 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.645 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.649 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.649 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.649 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.653 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:52.653 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.653 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.653 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:52.653 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:52.654 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:52.654 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:07:52.654 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:07:52.702 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:07:52.702 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:07:52.702 INFO 2458 --- [ocp-schedules-7,aca505108ddf639b,ecd27d1068480ab4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:07:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:07:53.220 INFO 2458 --- [ocp-schedules-11,4a18ad6068b1cace,61001c81ce61eacb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20304,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:07:53.220 INFO 2458 --- [ocp-schedules-11,4a18ad6068b1cace,61001c81ce61eacb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:07:53.592 INFO 2458 --- [ocp-schedules-2,8d6e1c1b5028271d,b412cd3e91876518] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:07:53.596 INFO 2458 --- [ocp-schedules-2,8d6e1c1b5028271d,b412cd3e91876518] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:07:53.707 INFO 2458 --- [prometheus-minute-schedule-1,bcb913da42b2877c,80fa28592fff9899] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:53.715 INFO 2458 --- [prometheus-minute-schedule-1,bcb913da42b2877c,80fa28592fff9899] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:53.988 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:53.989 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:07:54.418 INFO 2458 --- [ocp-schedules-6,ab65dd16c3ceb07f,d6be45fd699943a3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:07:54.504 INFO 2458 --- [ocp-async-1,d36d0e3f52437272,427387ba92cc389d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:54.504 WARN 2458 --- [ocp-async-8,bb768cc6a5cc572f,5832f755edcc66d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:55.852 INFO 2458 --- [ocp-schedules-13,f432bac6b6f914a3,1bb45105a5ee1a70] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:07:56.504 WARN 2458 --- [ocp-async-9,a3d86e557ed5f567,86715354411b0b45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:56.504 INFO 2458 --- [ocp-async-5,cfdea47684b2d86b,8b5e244698a41e3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:56.707 INFO 2458 --- [prometheus-minute-schedule-1,8f0518a3f99acc1b,b53bdbf83a19f9ea] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:56.715 INFO 2458 --- [prometheus-minute-schedule-1,8f0518a3f99acc1b,b53bdbf83a19f9ea] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:56.989 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:56.990 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:07:57.615 INFO 2458 --- [ocp-schedules-26,3b68048b6c879a2c,32e08ee7c1587db8] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:07:57.640 INFO 2458 --- [ocp-schedules-26,3b68048b6c879a2c,32e08ee7c1587db8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:57.690 INFO 2458 --- [ocp-schedules-29,368b4a7ca24763de,0d5706d7a5f1e5a5] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:07:57.694 INFO 2458 --- [ocp-schedules-29,368b4a7ca24763de,0d5706d7a5f1e5a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:07:57.734 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:07:57.734 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.735 INFO 2458 --- [ocp-schedules-29,368b4a7ca24763de,0d5706d7a5f1e5a5] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:07:57.738 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:07:57.746 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.747 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.773 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.773 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.786 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.786 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.787 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:07:57.787 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.787 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.787 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.790 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:07:57.799 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.799 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.801 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.802 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.807 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.808 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.808 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.808 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.829 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.830 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.837 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.837 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.837 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.837 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.854 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.855 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.864 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.864 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.865 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.865 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:07:57.867 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.867 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.867 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.868 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.889 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.890 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:57.891 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.892 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:57.892 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.893 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:57.894 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.895 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:57.895 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.896 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:57.897 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.897 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.897 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.897 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.897 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.898 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:57.898 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:07:57.901 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:07:57.924 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.924 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.924 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.925 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:07:57.927 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.927 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.927 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.928 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:07:57.953 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:57.954 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.954 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:07:57.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.963 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.963 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.964 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:07:57.964 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:57.964 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:07:57.964 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:57.964 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:07:57.967 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:07:57.967 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:07:57.995 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.995 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.995 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.995 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:07:57.998 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:57.998 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:57.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:57.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:07:58.027 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.027 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.027 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.028 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:07:58.054 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.054 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.054 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.055 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:07:58.101 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.101 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.102 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.103 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:58.103 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.103 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.103 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.104 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:58.105 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.105 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.105 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:58.106 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.106 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.107 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:58.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.108 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:58.109 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.109 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.110 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:58.111 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.111 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.111 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.112 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:58.112 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.112 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.112 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.113 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:58.113 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.113 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.113 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.114 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:58.114 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.115 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.115 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.115 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:07:58.116 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.116 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.116 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.117 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:07:58.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.117 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.119 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:07:58.119 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:07:58.122 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:07:58.198 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:07:58.198 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.198 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.199 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:07:58.199 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:07:58.199 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:07:58.200 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:07:58.224 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:07:58.228 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:07:58.228 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:07:58.229 INFO 2458 --- [ocp-schedules-26,3b68048b6c879a2c,32e08ee7c1587db8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:07:58.505 INFO 2458 --- [ocp-async-4,d1bd8765b1721686,d6f0dea908b41256] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:07:58.505 WARN 2458 --- [ocp-async-10,d7bd319e559d11c4,ade93d3e308714dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:07:58.509 INFO 2458 --- [ocp-async-2,2a35308caeefe40c,e9137285922ccae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:07:59.186 INFO 2458 --- [ocp-schedules-20,d7ad59847fa2fabe,9a80af9c86bd3355] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:07:59.707 INFO 2458 --- [prometheus-minute-schedule-1,0c5fb26a8d59b917,01adbf0dba0135c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534480, interval=60 2025-10-27 11:07:59.715 INFO 2458 --- [prometheus-minute-schedule-1,0c5fb26a8d59b917,01adbf0dba0135c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:07:59.990 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:07:59.991 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:00.505 WARN 2458 --- [ocp-async-1,5acc1aeb2290d5f1,d1c498890aaf089a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:00.505 INFO 2458 --- [ocp-async-3,decd59a105712eac,28ec15948feca8b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:01.715038+08:00, lastProcessTime=2025-10-27T11:07:51.775886+08:00 2025-10-27 11:08:01.924 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:01.715038+08:00 2025-10-27 11:08:01.954 INFO 2458 --- [ocp-schedules-8,83462c5e4805d804,27ed16ebcd8b9ad1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:02.056 WARN 2458 --- [scanner-112135278-1,83462c5e4805d804,881ad33f0b85c07c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:02.119 INFO 2458 --- [ocp-schedules-9,d69c0bb310cfede3,61f7ec1bb4353062] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:08:02.124 INFO 2458 --- [ocp-schedules-9,d69c0bb310cfede3,61f7ec1bb4353062] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:02.174 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:02.175 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:08:02.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:02.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:02.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:02.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:02.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:02.208 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:08:02.238 INFO 2458 --- [ocp-schedules-9,d69c0bb310cfede3,61f7ec1bb4353062] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:08:02.301 INFO 2458 --- [alarm-detect-9,83462c5e4805d804,5a938e853fdd373d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534450 ~ 1761534472 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534465, value=57.871)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534465, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534465=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:08:02.505 WARN 2458 --- [ocp-async-5,9f9945947e62433d,97118d166b381427] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:02.505 INFO 2458 --- [ocp-async-7,0c96319154159564,6944e89f995186d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:02.707 INFO 2458 --- [prometheus-minute-schedule-1,98ed3dc208db68cd,13d314d58f9338a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:02.716 INFO 2458 --- [prometheus-minute-schedule-1,98ed3dc208db68cd,13d314d58f9338a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:02.991 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:02.992 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:03.604 INFO 2458 --- [ocp-schedules-30,d2b416f512daf508,435d06445e76502c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:03.608 INFO 2458 --- [ocp-schedules-30,d2b416f512daf508,435d06445e76502c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:04.505 WARN 2458 --- [ocp-async-4,fd4d74965a763667,64ca85c1af30e0d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:04.505 INFO 2458 --- [ocp-async-4,267e92439d620b12,73166a9e60b395fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:05.707 INFO 2458 --- [prometheus-minute-schedule-1,20b21c991d67d6f1,339b4e417809abfb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:05.715 INFO 2458 --- [prometheus-minute-schedule-1,20b21c991d67d6f1,339b4e417809abfb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:05.860 INFO 2458 --- [ocp-schedules-21,6c69150891512156,30058e73de876f4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:05.992 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:05.993 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:08:06.056 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:08:06.098 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:08:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:08:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:08:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:08:06.121 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:08:06.123 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:08:06.124 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:06.125 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:08:06.127 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:06.127 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:08:06.312 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:05:30, start=2025-10-27T11:02:30, end=2025-10-27T11:05:00, updates=221, granule=120000ms, delay=180000ms 2025-10-27 11:08:06.320 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:05:30, target=2025-10-27T11:05:00, granule=120000ms, delay=30000ms) 2025-10-27 11:08:06.320 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:05:00, target=2025-10-27T10:52:30, granule=600000ms, delay=750000ms) 2025-10-27 11:08:06.320 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=322ms, period=120000ms 2025-10-27 11:08:06.505 INFO 2458 --- [ocp-async-8,05b6937998bd2d16,7643e7dd8b4af252] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:06.505 WARN 2458 --- [ocp-async-3,c531116ff6b5a43d,d92be90592b18de7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:06.696 INFO 2458 --- [ocp-schedules-13,a127ec16de2ec44f,449e728a21df5122] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:08:06.704 INFO 2458 --- [ocp-schedules-13,a127ec16de2ec44f,449e728a21df5122] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:06.704 INFO 2458 --- [ocp-schedules-13,a127ec16de2ec44f,449e728a21df5122] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:08.505 WARN 2458 --- [ocp-async-9,e1396446ae394ac6,24e91b5694befbb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:08.505 INFO 2458 --- [ocp-async-7,31d845bd778c78ac,ed122bcd4cc78dd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:08.510 INFO 2458 --- [ocp-async-10,3cd42227f39d444b,016a2b707e82f565] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:08.707 INFO 2458 --- [prometheus-minute-schedule-1,31fb09f0bfe1a027,12c06f3a8312feec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:08.715 INFO 2458 --- [prometheus-minute-schedule-1,31fb09f0bfe1a027,12c06f3a8312feec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:08.993 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:08.994 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:10.013 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:08:10.013 INFO 2458 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:08:10.048 INFO 2458 --- [Check obproxy1,797180e479d736c3,4791f3a54dc30dd3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:08:10.048 INFO 2458 --- [Check obproxy1,797180e479d736c3,4791f3a54dc30dd3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:08:10.062 INFO 2458 --- [Collect obproxy1,ef6748b7c8afe0ed,58ea5a8538a9ff00] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:08:10.062 INFO 2458 --- [Collect obproxy1,ef6748b7c8afe0ed,58ea5a8538a9ff00] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:08:10.078 INFO 2458 --- [Check obproxy1,797180e479d736c3,4791f3a54dc30dd3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:08:10.081 INFO 2458 --- [Check obproxy1,797180e479d736c3,4791f3a54dc30dd3] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:08:10.081 INFO 2458 --- [Check obproxy1,797180e479d736c3,4791f3a54dc30dd3] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:08:10.094 INFO 2458 --- [Collect obproxy1,ef6748b7c8afe0ed,58ea5a8538a9ff00] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:08:10.097 INFO 2458 --- [Collect obproxy1,ef6748b7c8afe0ed,58ea5a8538a9ff00] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:08:10.099 INFO 2458 --- [Collect obproxy1,ef6748b7c8afe0ed,58ea5a8538a9ff00] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:08:10.179 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:08:10.505 WARN 2458 --- [ocp-async-6,12cbf1416ebc0a48,ec0483daf566356a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:10.505 INFO 2458 --- [ocp-async-4,4ca2a03c02659e9c,2321d760fca1945b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:11.707 INFO 2458 --- [prometheus-minute-schedule-1,cef27dfd25353505,8fcae089ed328917] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:11.714 INFO 2458 --- [prometheus-minute-schedule-1,cef27dfd25353505,8fcae089ed328917] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:11.713126+08:00, lastProcessTime=2025-10-27T11:08:01.715038+08:00 2025-10-27 11:08:11.950 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:11.713126+08:00 2025-10-27 11:08:11.994 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:11.994 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:12.310 INFO 2458 --- [ocp-schedules-18,976955429cf3e3d9,ca89dc9506056ed6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:12.346 WARN 2458 --- [scanner-112137060-1,976955429cf3e3d9,90ebab08908bdc7a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:12.417 INFO 2458 --- [alarm-detect-9,976955429cf3e3d9,6b42ee481af1534a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534465 ~ 1761534482 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534480, value=53.254)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534480, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534480=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:08:12.506 INFO 2458 --- [ocp-async-3,4b131444c47f4c27,995509ed69c12fbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:12.506 WARN 2458 --- [ocp-async-1,275677b799436dc7,003f824c8e76216a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:13.616 INFO 2458 --- [ocp-schedules-3,431607e98e6a0637,5cfeccbb190d6bc4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:13.620 INFO 2458 --- [ocp-schedules-3,431607e98e6a0637,5cfeccbb190d6bc4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:13.686 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534420, nodeCount=817, affectRows=817 2025-10-27 11:08:13.709 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702} 2025-10-27 11:08:13.717 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocp 2025-10-27 11:08:13.718 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:13.719 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:13.719 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:08:13.719 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:13.722 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:13.727 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:13.727 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:08:13.727 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:13.732 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:13.734 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:13.734 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:08:13.734 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:13.737 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:13.738 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:13.738 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:08:13.738 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:13.740 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:13.757 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 17 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:13.796 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:08:13.796 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:13.917 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448093917000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=7F9C62DFE940539231935057E4965221, obServerId=null, obDbId=201009, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:08:13.917Z, endTime=2025-10-27T03:06:13.702Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:08:13.923 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448093923000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=549981E00A3D95F03DD5A9EF6044AA20, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:08:13.923Z, endTime=2025-10-27T03:06:13.702Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:08:13.930 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:08:13.930 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:08:13.931 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:08:13.931 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:08:13.959 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:13.959 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201009, sqlId=7F9C62DFE940539231935057E4965221, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:13.961 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:13.961 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:13.965 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:08:13.965 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:08:13.969 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:08:13.969 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:08:13.992 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:13.992 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201009, sqlId=7F9C62DFE940539231935057E4965221, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:13.993 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:13.993 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=549981E00A3D95F03DD5A9EF6044AA20, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:13.996 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:08:13.996 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:08:13.917, end=2025-10-27T11:06:13.702}, clusterId=2, tenantId=5, sqlId=7F9C62DFE940539231935057E4965221, serverId=null, dbName=NO_database, obDbId=201009, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:08:13.997 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:08:13.997 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:08:13.923, end=2025-10-27T11:06:13.702}, clusterId=2, tenantId=5, sqlId=549981E00A3D95F03DD5A9EF6044AA20, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:08:14.204 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448094204000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:08:14.204Z, endTime=2025-10-27T03:06:13.702Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:08:14.216 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:08:14.216 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:08:14.223 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448094223000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=5EBC5359C07B13CB559FB4845DB6767A, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:08:14.223Z, endTime=2025-10-27T03:06:13.702Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:08:14.234 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2025-10-27 11:08:14.234 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:08:14.238 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:14.238 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:14.244 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:08:14.244 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:08:14.257 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:14.257 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=5EBC5359C07B13CB559FB4845DB6767A, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:14.264 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:08:14.264 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:08:14.269 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:14.269 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:14.273 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:08:14.273 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:08:14.204, end=2025-10-27T11:06:13.702}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:08:14.288 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:08:14.288 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534373702000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=5EBC5359C07B13CB559FB4845DB6767A, sqlIdList=null, dynamicSql=false) 2025-10-27 11:08:14.292 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:08:14.320 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:08:14.223, end=2025-10-27T11:06:13.702}, clusterId=2, tenantId=5, sqlId=5EBC5359C07B13CB559FB4845DB6767A, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:08:14.320 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:08:14.322 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:04:13.057, end=2025-10-27T11:06:13.702},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:08:14.322 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:08:14.322 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:08:14.325 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:08:14.348 INFO 2458 --- [ocp-schedules-25,8c2320dc4ba70c88,f69f6620bac3b3c6] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:08:14.348+08:00, clusters=[1, 2] 2025-10-27 11:08:14.349 INFO 2458 --- [ocp-schedules-25,8c2320dc4ba70c88,f69f6620bac3b3c6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:08:14.351 INFO 2458 --- [ocp-schedules-27,92201df2cd470693,decbc14f5a6e3585] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:08:14.351 INFO 2458 --- [ocp-schedules-25,8c2320dc4ba70c88,f69f6620bac3b3c6] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:08:14.351 INFO 2458 --- [ocp-schedules-27,92201df2cd470693,decbc14f5a6e3585] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:08:14.410 INFO 2458 --- [Collect all tenants major compactions1,92201df2cd470693,108e53bb88c06e85] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:08:14.410 INFO 2458 --- [Collect all tenants major compactions4,92201df2cd470693,6a4a17f54c925c22] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:08:14.410 INFO 2458 --- [Collect all tenants major compactions2,92201df2cd470693,51a4cbb885c85bbc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:08:14.411 INFO 2458 --- [Collect all tenants major compactions3,92201df2cd470693,1c85d97b8d601c75] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:08:14.411 INFO 2458 --- [Collect all tenants major compactions6,92201df2cd470693,c1a63455b8cdbb76] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:08:14.411 INFO 2458 --- [Collect all tenants major compactions5,92201df2cd470693,34a42db7086deb30] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:08:14.443 INFO 2458 --- [Collect all tenants major compactions5,92201df2cd470693,34a42db7086deb30] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.445 INFO 2458 --- [Collect all tenants major compactions5,92201df2cd470693,34a42db7086deb30] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:08:14.445 INFO 2458 --- [Collect all tenants major compactions6,92201df2cd470693,c1a63455b8cdbb76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.447 INFO 2458 --- [Collect all tenants major compactions6,92201df2cd470693,c1a63455b8cdbb76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:08:14.448 INFO 2458 --- [Collect all tenants major compactions5,92201df2cd470693,34a42db7086deb30] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:08:14.453 INFO 2458 --- [Collect all tenants major compactions6,92201df2cd470693,c1a63455b8cdbb76] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:08:14.458 INFO 2458 --- [Collect all tenants major compactions4,92201df2cd470693,6a4a17f54c925c22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.459 INFO 2458 --- [Collect all tenants major compactions4,92201df2cd470693,6a4a17f54c925c22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:08:14.465 INFO 2458 --- [Collect all tenants major compactions4,92201df2cd470693,6a4a17f54c925c22] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:08:14.467 INFO 2458 --- [Collect all tenants major compactions2,92201df2cd470693,51a4cbb885c85bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.467 INFO 2458 --- [Collect all tenants major compactions3,92201df2cd470693,1c85d97b8d601c75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.468 INFO 2458 --- [Collect all tenants major compactions1,92201df2cd470693,108e53bb88c06e85] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:14.468 INFO 2458 --- [Collect all tenants major compactions2,92201df2cd470693,51a4cbb885c85bbc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:08:14.469 INFO 2458 --- [Collect all tenants major compactions3,92201df2cd470693,1c85d97b8d601c75] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:08:14.470 INFO 2458 --- [Collect all tenants major compactions1,92201df2cd470693,108e53bb88c06e85] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:08:14.471 INFO 2458 --- [Collect all tenants major compactions2,92201df2cd470693,51a4cbb885c85bbc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:08:14.473 INFO 2458 --- [Collect all tenants major compactions1,92201df2cd470693,108e53bb88c06e85] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:08:14.473 INFO 2458 --- [Collect all tenants major compactions3,92201df2cd470693,1c85d97b8d601c75] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:08:14.474 INFO 2458 --- [ocp-schedules-27,92201df2cd470693,decbc14f5a6e3585] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:08:14.506 WARN 2458 --- [ocp-async-9,3d1b76b4f0d84f0f,8ec4980568eb24ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:14.506 INFO 2458 --- [ocp-async-9,77817933b538d616,5128a7238bf5b671] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:14.651 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=1000, affectRows=1000 2025-10-27 11:08:14.656 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=994, validCount=992 2025-10-27 11:08:14.701 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=1000, affectRows=1000 2025-10-27 11:08:14.706 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=999, validCount=995 2025-10-27 11:08:14.707 INFO 2458 --- [prometheus-minute-schedule-1,d2b4aa10e5bc5a91,b53ebe96e6056530] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:14.717 INFO 2458 --- [prometheus-minute-schedule-1,d2b4aa10e5bc5a91,b53ebe96e6056530] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:14.721 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134521640) Duplicate entry '835370603008958464-1761534360' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134521640) Duplicate entry '835370603008958464-1761534360' for key 'PRIMARY' 2025-10-27 11:08:14.940 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=1000, affectRows=1000 2025-10-27 11:08:14.945 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=1000, validCount=996 2025-10-27 11:08:14.995 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:14.995 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=77, affectRows=77 2025-10-27 11:08:14.995 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=76, validCount=76 2025-10-27 11:08:14.995 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:14.998 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=1000, affectRows=1000 2025-10-27 11:08:15.002 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=996, validCount=992 2025-10-27 11:08:15.097 INFO 2458 --- [ocp-schedules-12,e83968987d86209d,6b3012cfe1a00bf2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:08:15.098 INFO 2458 --- [ocp-schedules-12,e83968987d86209d,6b3012cfe1a00bf2] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:08:15.117 INFO 2458 --- [ocp-schedules-12,e83968987d86209d,6b3012cfe1a00bf2] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:08:15.117 INFO 2458 --- [ocp-schedules-12,e83968987d86209d,6b3012cfe1a00bf2] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:08:15.867 INFO 2458 --- [ocp-schedules-14,ef08a499c7c5702b,e6c3295e614fbee5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:16.506 INFO 2458 --- [ocp-async-6,f147392d7f2a3f6b,4ef6fb421b019e76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:16.506 WARN 2458 --- [ocp-async-2,bcc96a6a6a486cc0,ed663d6be4db040f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:17.707 INFO 2458 --- [prometheus-minute-schedule-1,0dd8535a26094afe,933a480acba032bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:17.715 INFO 2458 --- [prometheus-minute-schedule-1,0dd8535a26094afe,933a480acba032bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:17.995 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:17.996 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:18.506 WARN 2458 --- [ocp-async-4,2691b056a979b1dc,06f6a9c59dbb2867] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:18.506 INFO 2458 --- [ocp-async-4,137cc5de2f058bb4,f2cad427719ce229] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:18.510 INFO 2458 --- [ocp-async-3,7cfdeed67611a5d4,19f2b2d9e7df0b01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:19.223 INFO 2458 --- [ocp-schedules-6,3ef148209ca9f0a6,4e3dcb0ba06ff8c0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:08:19.655 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534420, nodeCount=1000, affectRows=1000 2025-10-27 11:08:19.659 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534420, nodeCount=995, validCount=982 2025-10-27 11:08:20.506 WARN 2458 --- [ocp-async-3,62c0112cf655f62e,09465b37709e2a90] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:20.506 INFO 2458 --- [ocp-async-7,b97effd3591f359c,29c1ed229b3cfcc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:20.707 INFO 2458 --- [prometheus-minute-schedule-1,b9adb8bbeb785233,602df93ce10c0ba2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:20.722 INFO 2458 --- [prometheus-minute-schedule-1,b9adb8bbeb785233,602df93ce10c0ba2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:20.996 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:20.997 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:21.413 INFO 2458 --- [ocp-schedules-30,d690735f543a260e,79c32100af5c3541] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:21.413 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:08:21.413 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:08:21.413 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:08:21.414 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:08:21.414 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:08:21.429 INFO 2458 --- [ocp-schedules-25,d75b2f7acac01b25,c6201df53d081c2f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:21.429 INFO 2458 --- [host-check-clockdiff1,d75b2f7acac01b25,f1153e768c96a834] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:08:21.429 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:21.429 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:21.430 INFO 2458 --- [host-check-clockdiff2,d75b2f7acac01b25,00f5f4454705400c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:08:21.430 INFO 2458 --- [host-check-clockdiff3,d75b2f7acac01b25,ec517c5237fdc72e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:08:21.430 INFO 2458 --- [host-check-clockdiff4,d75b2f7acac01b25,2e53e127dc1da10d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:08:21.431 INFO 2458 --- [host-check-clockdiff5,d75b2f7acac01b25,e256f4917a062b40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:08:21.431 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:21.445 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:21.445 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:21.451 INFO 2458 --- [ocp-schedules-4,686a63c805eb3039,a7fc3feedde863be] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:08:21.451 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:21.453 INFO 2458 --- [host-check-clockdiff4,d75b2f7acac01b25,2e53e127dc1da10d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534501 -1 -1 , error= 2025-10-27 11:08:21.453 INFO 2458 --- [host-check-clockdiff5,d75b2f7acac01b25,e256f4917a062b40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534501 0 0 , error= 2025-10-27 11:08:21.457 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:21.458 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:21.458 INFO 2458 --- [host-check-clockdiff1,d75b2f7acac01b25,f1153e768c96a834] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534501 0 0 , error= 2025-10-27 11:08:21.453 INFO 2458 --- [ocp-schedules-29,633b01abd4860bd1,3c4d6dbcbc2030a7] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:21.458 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:08:21.458 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:08:21.458 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:08:21.463 INFO 2458 --- [host-check-clockdiff2,d75b2f7acac01b25,00f5f4454705400c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534501 -1 -1 , error= 2025-10-27 11:08:21.464 INFO 2458 --- [host-check-clockdiff3,d75b2f7acac01b25,ec517c5237fdc72e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534501 0 0 , error= 2025-10-27 11:08:21.467 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:21.467 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:21.468 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:08:21.468 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:08:21.485 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:21.485 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:21.487 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:21.487 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:21.487 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:21.488 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:21.488 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:21.490 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:21.490 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:21.492 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:21.544 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:08:21.545 INFO 2458 --- [ocp-schedules-14,b5cf7fb91ed3d644,ebebe3bbc1af625a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:08:21.548 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:08:21.549 INFO 2458 --- [ocp-schedules-13,d4b97d344e0d9e40,c16bd5ac0cb71998] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:08:21.549 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:08:21.549 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:08:21.549 INFO 2458 --- [host-check-agent-status1,d690735f543a260e,99eae38eb9b9e504] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:08:21.549 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:08:21.550 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:08:21.551 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:08:21.551 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:08:21.551 INFO 2458 --- [host-check-agent-status5,d690735f543a260e,15b05c55c62d2323] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:08:21.553 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:08:21.553 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:08:21.553 INFO 2458 --- [host-check-agent-status4,d690735f543a260e,e6a67e9368cd06f3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:08:21.554 INFO 2458 --- [ocp-schedules-7,ab4ac2f7c65ed062,3bb98fc40b961de0] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:08:21.554 INFO 2458 --- [ocp-schedules-14,b5cf7fb91ed3d644,ebebe3bbc1af625a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:21.554 INFO 2458 --- [ocp-schedules-13,d4b97d344e0d9e40,c16bd5ac0cb71998] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:21.555 INFO 2458 --- [ocp-schedules-13,d4b97d344e0d9e40,c16bd5ac0cb71998] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:08:21.554 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:08:21.555 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:08:21.555 INFO 2458 --- [host-check-agent-status3,d690735f543a260e,e232c4eaaab79e6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:08:21.561 INFO 2458 --- [ocp-schedules-7,ab4ac2f7c65ed062,3bb98fc40b961de0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:21.562 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:08:21.562 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:08:21.562012455+08:00 2025-10-27 11:08:21.563 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:08:21.562+08:00 2025-10-27 11:08:21.564 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:08:21.566 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:08:21.565356773+08:00 2025-10-27 11:08:21.567 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:08:21.566+08:00 2025-10-27 11:08:21.575 INFO 2458 --- [ocp-schedules-16,c835ccf6b0f088e4,a7505a107c980ccd] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:08:21.578 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:08:21.562+08:00 and status AVAILABLE 2025-10-27 11:08:21.579 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:08:21.566+08:00 and status AVAILABLE 2025-10-27 11:08:21.579 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:08:21.580 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:08:21.580124067+08:00 2025-10-27 11:08:21.581 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:08:21.580+08:00 2025-10-27 11:08:21.582 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:08:21.582 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:08:21.582806855+08:00 2025-10-27 11:08:21.583 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:08:21.583 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:08:21.582965786+08:00 2025-10-27 11:08:21.584 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:08:21.582+08:00 2025-10-27 11:08:21.584 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:08:21.583+08:00 2025-10-27 11:08:21.585 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:08:21.587 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:08:21.589 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:08:21.580+08:00 and status AVAILABLE 2025-10-27 11:08:21.594 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:08:21.583+08:00 and status AVAILABLE 2025-10-27 11:08:21.595 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:08:21.595 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:08:21.582+08:00 and status AVAILABLE 2025-10-27 11:08:21.602 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:08:21.604 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:08:21.605 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:08:21.608 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:08:21.608 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:08:21.608 INFO 2458 --- [host-check-agent-status2,d690735f543a260e,9e733352eaec5d45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:08:21.613 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.614 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:08:21.619 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:21.623 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:21.629 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:21.632 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:21.633 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:21.633 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:21.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.635 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.636 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:21.636 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:21.637 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:21.637 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:21.638 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:21.642 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:08:21.642 INFO 2458 --- [ocp-schedules-14,b5cf7fb91ed3d644,ebebe3bbc1af625a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:08:21.653 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.654 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:21.691 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:21.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:21.705 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:21.707 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:21.710 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:21.711 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:21.711 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:08:21.722 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:08:21.727 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:08:21.728 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:08:21.740 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:08:21.741 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:21.742 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.743 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:08:21.760 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:21.758441+08:00, lastProcessTime=2025-10-27T11:08:11.713126+08:00 2025-10-27 11:08:21.761 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:08:21.762 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:08:21.766 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:08:21.766 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:08:21.771 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:08:21.796 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:21.802 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:21.803 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.803 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:21.804 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.805 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:21.806 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.806 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:08:21.806 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:21.808 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:08:21.811 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:08:21.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:21.814 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:21.823 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:08:21.826 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:08:21.830 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:08:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:21.841 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:08:21.845 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:08:21.861 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:08:21.862 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:08:21.865 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:08:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:21.883 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:08:21.886 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:21.899 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:21.901 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:21.901 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:08:21.904 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.916 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:08:21.916 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:21.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:21.917 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:21.918 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:21.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:08:21.920 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:08:21.921 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:08:21.922 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:21.923 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:08:21.944 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:08:21.949 INFO 2458 --- [host-check-host-status4,633b01abd4860bd1,59f3da2f27798d75] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:08:21.956 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:21.957 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:08:21.959 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:21.962 INFO 2458 --- [host-check-host-status5,633b01abd4860bd1,223a403da70c3af7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:08:21.966 INFO 2458 --- [host-check-host-status2,633b01abd4860bd1,a8fa2731f3812df0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:08:21.974 INFO 2458 --- [host-check-host-status3,633b01abd4860bd1,4c948afc0f8f6ec8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:08:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:21.990 INFO 2458 --- [host-check-host-status1,633b01abd4860bd1,3032e1cf4ceb67f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:08:22.025 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:22.025 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:08:22.056 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:21.758441+08:00 2025-10-27 11:08:22.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:22.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:22.077 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:08:22.077 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:08:22.102 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:22.102 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:22.104 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.105 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:08:22.122 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:08:22.138 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:22.138 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:22.158 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:22.159 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.159 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:08:22.189 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:08:22.189 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:08:22.194 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:22.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:22.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.239 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.239 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.239 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.241 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:22.247 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:08:22.260 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:08:22.270 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:22.274 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:22.342 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:08:22.391 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:08:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:08:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.394 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.395 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:08:22.432 INFO 2458 --- [ocp-schedules-12,5021044cc1f88cf1,b8f9ebb32d62e5f7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:22.442 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:08:22.488 WARN 2458 --- [scanner-112137236-1,5021044cc1f88cf1,758e6c829b3f921d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:22.496 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:22.496 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.496 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.497 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:22.497 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:22.497 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:22.499 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:08:22.506 WARN 2458 --- [ocp-async-2,17b3ff68566229b8,c8db0b004e730198] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:22.506 INFO 2458 --- [ocp-async-7,a28a210ecd44160b,5902bbc69cb366f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:22.572 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:08:22.572 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:08:22.591 INFO 2458 --- [ocp-schedules-7,ab4ac2f7c65ed062,3bb98fc40b961de0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:08:23.632 INFO 2458 --- [ocp-schedules-11,aa5e135f80e01eb5,e9a854239c2bf98c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:23.636 INFO 2458 --- [ocp-schedules-11,aa5e135f80e01eb5,e9a854239c2bf98c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:23.707 INFO 2458 --- [prometheus-minute-schedule-1,9bc3f8e6410e4fc6,448daeaf4d54d7c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:23.720 INFO 2458 --- [prometheus-minute-schedule-1,9bc3f8e6410e4fc6,448daeaf4d54d7c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:23.997 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:23.999 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:24.506 INFO 2458 --- [ocp-async-7,b18132d6c3156166,cd49df22fa59ae47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:24.507 WARN 2458 --- [ocp-async-1,3a79795c5a08c82f,6757e3d0964f53e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:25.875 INFO 2458 --- [ocp-schedules-18,add7765ad6552a0f,84b31b42fdca4389] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:26.507 INFO 2458 --- [ocp-async-7,7027bbda38411425,4ff947cdb9ca1d61] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:26.507 WARN 2458 --- [ocp-async-9,33801e598d611d1b,5ab81dcf8cf69436] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:26.697 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:08:26.697 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:08:26.700 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:08:26.700 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:08:26.700 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:08:26.700 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:08:26.704 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:08:26.704 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:08:26.704 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:08:26.704 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:08:26.707 INFO 2458 --- [prometheus-minute-schedule-1,6e19ba93ac4be4dc,9f4885227907149b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:26.719 INFO 2458 --- [prometheus-minute-schedule-1,6e19ba93ac4be4dc,9f4885227907149b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:26.731 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-13,47d1d9e4d873119c,5a07fbfc5c051ad6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:08:26.736 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:08:26.743 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:08:26.753 INFO 2458 --- [ocp-schedules-13,47d1d9e4d873119c,5a07fbfc5c051ad6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:26.753 INFO 2458 --- [ocp-schedules-13,47d1d9e4d873119c,5a07fbfc5c051ad6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:26.999 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:27.002 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:27.105 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:08:27.496 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.791777784S 2025-10-27 11:08:27.496 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:08:27.565 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:08:27.565 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:08:27.604 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:08:27.653 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:08:27.653 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:08:27.653 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:08:27.653 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:08:27.653 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:08:27.654 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:08:27.661 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:08:27.779 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] o.s.boot.SpringApplication : Started application in 1.078 seconds (JVM running for 256976.206) 2025-10-27 11:08:27.779 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.078778394S 2025-10-27 11:08:28.120 INFO 2458 --- [ocp-schedules-4,b5a0ea5c54eab5af,8046b6313e720f3c] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:08:28.507 INFO 2458 --- [ocp-async-10,e2ebcb0f2e4abc98,9ab7be34d5b078f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:28.507 WARN 2458 --- [ocp-async-6,98e1381ef6cc8af7,cdc97d5015424087] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:28.510 INFO 2458 --- [ocp-async-2,4c157fa7dfd6088e,2b1d3fea36f35583] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:28.847 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:08:28.847 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:08:28.853 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:08:28.853 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:08:28.857 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:08:28.857 INFO 2458 --- [ocp-schedules-16,6d864db98ba28961,4937a3b857deaec0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:08:29.313 INFO 2458 --- [ocp-schedules-13,c9825d60373ec853,08a81c40fd36656a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:08:29.324 INFO 2458 --- [ocp-schedules-13,c9825d60373ec853,08a81c40fd36656a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:08:29.324 INFO 2458 --- [ocp-schedules-13,c9825d60373ec853,08a81c40fd36656a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:08:29.707 INFO 2458 --- [prometheus-minute-schedule-1,0110545598e48f18,ad91e3a3b3675551] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:29.718 INFO 2458 --- [prometheus-minute-schedule-1,0110545598e48f18,ad91e3a3b3675551] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:30.002 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:30.004 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:30.507 INFO 2458 --- [ocp-async-10,81220af1a12c087b,08521834739aa906] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:30.507 WARN 2458 --- [ocp-async-8,6f3c7360c731322e,6543f020dd4d4631] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:31.715489+08:00, lastProcessTime=2025-10-27T11:08:21.758441+08:00 2025-10-27 11:08:31.983 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:31.715489+08:00 2025-10-27 11:08:32.507 INFO 2458 --- [ocp-async-5,749c92d3413113dc,a89fbfefd3865f98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:32.507 WARN 2458 --- [ocp-async-3,14a40f65328ea359,5a6f565395a60c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:32.648 INFO 2458 --- [ocp-schedules-9,1cedc5395685f5dd,f7ccdc930a2551ff] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:32.729 WARN 2458 --- [scanner-112137500-1,1cedc5395685f5dd,6e8df72cddbc8c34] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:32.731 INFO 2458 --- [prometheus-minute-schedule-1,0fa3520055a0b090,3e49a7f8f41dd600] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:32.755 INFO 2458 --- [prometheus-minute-schedule-1,0fa3520055a0b090,3e49a7f8f41dd600] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:32.812 INFO 2458 --- [alarm-detect-13,1cedc5395685f5dd,bf47bf4422809dfa] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534480 ~ 1761534502 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534495, value=61.2)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534495, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534495=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:08:33.004 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:33.005 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:33.644 INFO 2458 --- [ocp-schedules-20,25019902ac5bf1a8,5d17f56a09266248] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:33.648 INFO 2458 --- [ocp-schedules-20,25019902ac5bf1a8,5d17f56a09266248] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:33.734 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5e5e555fd3bc75ca,e78bcf378c3626ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=5e5e555fd3bc75ca, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:08:33.734 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5e5e555fd3bc75ca,e78bcf378c3626ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=5e5e555fd3bc75ca, duration=2 ms] 2025-10-27 11:08:33.744 INFO 2458 --- [ocp-schedules-18,8433942d30f918c5,86f6fc0627635459] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:08:33.744 INFO 2458 --- [ocp-schedules-18,8433942d30f918c5,86f6fc0627635459] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:08:34.507 WARN 2458 --- [ocp-async-7,32e3b8f09f32725d,f0848a019da1e231] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:34.507 INFO 2458 --- [ocp-async-7,dcfc831b32cfb22b,6d2033b67c1f8b82] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:35.121 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:08:35.121 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:08:35.246 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:35.246 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:35.289 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:35.289 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:35.290 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:35.291 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:08:35.305 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:35.305 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:35.305 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:35.306 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:08:35.369 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:35.370 INFO 2458 --- [ocp-schedules-16,f10452a13831d5d4,b778eb7e5ff9718a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:08:35.707 INFO 2458 --- [prometheus-minute-schedule-1,2b7d460d3a7cd2e1,5b8432f5d358544b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:35.716 INFO 2458 --- [prometheus-minute-schedule-1,2b7d460d3a7cd2e1,5b8432f5d358544b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:35.882 INFO 2458 --- [ocp-schedules-22,7ee18090e09fe4b6,7a46fcbb78e8f460] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:36.005 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:36.006 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:36.507 INFO 2458 --- [ocp-async-8,d8177f031909f8b7,d1475913bd1866b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:36.507 WARN 2458 --- [ocp-async-4,80458ddcbdfb5fa6,4251611b8324c248] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:37.056 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:37.056 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:08:37.056 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:08:37.056 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:08:37.056 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:08:37.056 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:08:37.071 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:37.071 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:37.071 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:37.071 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:37.075 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:37.075 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:37.077 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:37.077 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:37.086 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:37.089 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:37.210 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:08:37.215 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:08:37.220 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:08:37.230 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:08:37.238 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:08:37.239 INFO 2458 --- [ocp-schedules-29,70291e68b0fe504a,77d41e93b0e05937] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:08:37.708 INFO 2458 --- [prometheus-cache-monitor1,790b335f5de92b55,cf398e2baf9dda65] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20306,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:08:37.708 INFO 2458 --- [prometheus-cache-monitor1,790b335f5de92b55,cf398e2baf9dda65] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:08:38.507 WARN 2458 --- [ocp-async-10,df7728ba1bc4cebc,fcf02ce88315f446] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:38.507 INFO 2458 --- [ocp-async-3,af78c767c7746e82,3fdb85386a7ad9e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:38.510 INFO 2458 --- [ocp-async-5,10d9c551f8fdaeb2,c0e66690ed197102] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:38.593 INFO 2458 --- [ocp-schedules-7,d057387dfb4f5771,a6168aed72f99680] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:08:38.602 INFO 2458 --- [ocp-schedules-7,d057387dfb4f5771,a6168aed72f99680] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:38.603 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:08:38.603 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:08:38.603 INFO 2458 --- [ocp-schedules-7,d057387dfb4f5771,a6168aed72f99680] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:08:38.707 INFO 2458 --- [prometheus-minute-schedule-1,ea9908279ffeef7c,d1d32480a600b882] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:38.715 INFO 2458 --- [prometheus-minute-schedule-1,ea9908279ffeef7c,d1d32480a600b882] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:39.006 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:39.008 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:40.049 INFO 2458 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:08:40.049 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:08:40.112 INFO 2458 --- [Collect obproxy1,28971315310ce40a,e5440f84a59023c8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:08:40.112 INFO 2458 --- [Check obproxy1,be141bc76942b18e,79bccf3459470290] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:08:40.117 INFO 2458 --- [Collect obproxy1,28971315310ce40a,e5440f84a59023c8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:08:40.118 INFO 2458 --- [Check obproxy1,be141bc76942b18e,79bccf3459470290] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:08:40.147 INFO 2458 --- [Collect obproxy1,28971315310ce40a,e5440f84a59023c8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:08:40.149 INFO 2458 --- [Check obproxy1,be141bc76942b18e,79bccf3459470290] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:08:40.150 INFO 2458 --- [Collect obproxy1,28971315310ce40a,e5440f84a59023c8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:08:40.152 INFO 2458 --- [Check obproxy1,be141bc76942b18e,79bccf3459470290] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:08:40.152 INFO 2458 --- [Check obproxy1,be141bc76942b18e,79bccf3459470290] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:08:40.153 INFO 2458 --- [Collect obproxy1,28971315310ce40a,e5440f84a59023c8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:08:40.508 INFO 2458 --- [ocp-async-6,f987931a8a5bd101,403c7c9abb6e2ed8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:40.508 WARN 2458 --- [ocp-async-7,d0291a956b2cdcfb,58d85d58dd6f4fdc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:40.526 INFO 2458 --- [ocp-schedules-9,45eff5f8435a0b4f,8da79fa1e4197f89] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:08:40.526 INFO 2458 --- [ocp-schedules-9,45eff5f8435a0b4f,8da79fa1e4197f89] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:08:40.551 INFO 2458 --- [ocp-schedules-9,45eff5f8435a0b4f,8da79fa1e4197f89] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:08:41.707 INFO 2458 --- [prometheus-minute-schedule-1,6af5ebc3056f7e41,a29d23747f6b4916] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:41.714 INFO 2458 --- [prometheus-minute-schedule-1,6af5ebc3056f7e41,a29d23747f6b4916] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:41.714036+08:00, lastProcessTime=2025-10-27T11:08:31.715489+08:00 2025-10-27 11:08:41.919 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:41.714036+08:00 2025-10-27 11:08:42.008 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:42.009 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:42.508 INFO 2458 --- [ocp-async-4,6e74a0f9c4b8fb1d,5fbd840abae0f141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:42.508 WARN 2458 --- [ocp-async-8,232ef5ca55334134,1ece538a86a87f20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:42.819 INFO 2458 --- [ocp-schedules-14,182c4852df81179e,6f71c3ad01bca713] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:42.879 WARN 2458 --- [scanner-112137704-1,182c4852df81179e,5148fe00a536d450] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:42.995 INFO 2458 --- [alarm-detect-20,182c4852df81179e,71241a7a4c1fe7d8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534495 ~ 1761534512 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534510, value=63.463)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534510, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534510=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:08:43.658 INFO 2458 --- [ocp-schedules-30,a92581d09555201a,71a36eef1f3958db] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:43.662 INFO 2458 --- [ocp-schedules-30,a92581d09555201a,71a36eef1f3958db] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:43.993 INFO 2458 --- [ocp-schedules-3,ae0eedfc650c47b7,210ca4dd30c603dc] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:08:44.508 INFO 2458 --- [ocp-async-10,5a95eff82589c136,d6219d8f8178dfd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:44.508 WARN 2458 --- [ocp-async-5,ff617e62bf0ce9e7,b23b1a15a94e455f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:44.707 INFO 2458 --- [prometheus-minute-schedule-1,382b1f4e7174d6ed,8d24b1107a33b2d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:44.714 INFO 2458 --- [prometheus-minute-schedule-1,382b1f4e7174d6ed,8d24b1107a33b2d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:45.009 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:45.010 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:45.102 INFO 2458 --- [prometheus-second-schedule-1,553cc00f0417afeb,a41715c3f4c57543] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:08:45.288 INFO 2458 --- [ocp-schedules-27,25a8118ee7d0f1bd,2daaf3284ba28ac9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:08:45.288 INFO 2458 --- [ocp-schedules-27,25a8118ee7d0f1bd,2daaf3284ba28ac9] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:08:45.298 INFO 2458 --- [ocp-schedules-27,25a8118ee7d0f1bd,2daaf3284ba28ac9] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:08:45.298 INFO 2458 --- [ocp-schedules-27,25a8118ee7d0f1bd,2daaf3284ba28ac9] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:08:45.888 INFO 2458 --- [ocp-schedules-22,85971cb4a58a7385,e846966c5ec1d2c7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:46.508 WARN 2458 --- [ocp-async-7,cfcf062468ee278f,50282cfba8056a69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:46.508 INFO 2458 --- [ocp-async-1,b13525e0487d4ac4,850f95468f05c054] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:46.758 INFO 2458 --- [ocp-schedules-26,4576a2a9922e0b96,ea2bae122c8b28d7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:08:46.772 INFO 2458 --- [ocp-schedules-26,4576a2a9922e0b96,ea2bae122c8b28d7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:46.772 INFO 2458 --- [ocp-schedules-26,4576a2a9922e0b96,ea2bae122c8b28d7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:08:47.707 INFO 2458 --- [prometheus-minute-schedule-1,1f0c7a5217d1ac0a,7bc9b0de2f29c60c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:47.715 INFO 2458 --- [prometheus-minute-schedule-1,1f0c7a5217d1ac0a,7bc9b0de2f29c60c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:47.806 INFO 2458 --- [ocp-schedules-15,930a46de0326fe1e,4467a179a17fd890] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:08:47.806+08:00, clusters=[1, 2] 2025-10-27 11:08:47.821 INFO 2458 --- [ocp-schedules-15,930a46de0326fe1e,4467a179a17fd890] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:08:47.864 INFO 2458 --- [ocp-schedules-15,930a46de0326fe1e,4467a179a17fd890] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster myoceanbase2, exclude result is empty. 2025-10-27 11:08:48.010 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:48.012 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:48.508 WARN 2458 --- [ocp-async-8,bcb77b6421d6bfb5,901f962e3d76addf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:48.508 INFO 2458 --- [ocp-async-2,d7de51d03f3ef955,55c1afc69f81d922] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:48.510 INFO 2458 --- [ocp-async-4,ef1c08b679b3ed70,167228ed48759a67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:50.508 WARN 2458 --- [ocp-async-9,6ed44010fcbcf4bb,a8b69e7d28bce292] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:50.508 INFO 2458 --- [ocp-async-10,4f261e620682362c,fc47f2f9f75de961] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:50.701 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,378c4e73330ee6a4,36c08c7b57e658d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=378c4e73330ee6a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:08:50.701 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,378c4e73330ee6a4,36c08c7b57e658d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=378c4e73330ee6a4, duration=1 ms] 2025-10-27 11:08:50.707 INFO 2458 --- [prometheus-minute-schedule-1,55a10a8122c259fb,2c3bd4accbf87cae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:50.714 INFO 2458 --- [prometheus-minute-schedule-1,55a10a8122c259fb,2c3bd4accbf87cae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:51.012 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:51.013 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:51.442 INFO 2458 --- [ocp-schedules-7,356772c03c841bf2,f47c0950a727e3dd] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:08:51.442 INFO 2458 --- [ocp-schedules-4,aa8efeed78a2b9a8,f0d60ddb00accb55] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:51.442 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:08:51.442 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:08:51.442 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:08:51.443 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:08:51.443 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:08:51.447 INFO 2458 --- [ocp-schedules-19,71c7eda4fcce167d,dd3b4d46fd5b393d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:51.448 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:08:51.448 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:08:51.448 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:08:51.450 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:08:51.450 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:08:51.459 INFO 2458 --- [ocp-schedules-22,27d8a95d67640026,5ebf92983de00037] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:08:51.459 INFO 2458 --- [host-check-clockdiff1,27d8a95d67640026,ebc156f3b3bd1497] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:08:51.459 INFO 2458 --- [host-check-clockdiff2,27d8a95d67640026,cc436c442df20a2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:08:51.460 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:51.460 INFO 2458 --- [host-check-clockdiff5,27d8a95d67640026,14cfbe103e629228] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:08:51.460 INFO 2458 --- [host-check-clockdiff3,27d8a95d67640026,8e7bcb0ff021aa99] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:08:51.460 INFO 2458 --- [host-check-clockdiff4,27d8a95d67640026,1a1da30596b61328] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:08:51.461 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:51.462 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:51.468 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:51.469 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:51.485 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:51.485 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:51.490 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:51.490 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:51.490 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:51.491 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:51.498 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:51.498 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:51.499 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:51.499 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:51.501 INFO 2458 --- [host-check-clockdiff4,27d8a95d67640026,1a1da30596b61328] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534531 -1 -1 , error= 2025-10-27 11:08:51.501 INFO 2458 --- [host-check-clockdiff1,27d8a95d67640026,ebc156f3b3bd1497] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534531 0 0 , error= 2025-10-27 11:08:51.503 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:51.509 INFO 2458 --- [host-check-clockdiff5,27d8a95d67640026,14cfbe103e629228] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534531 0 0 , error= 2025-10-27 11:08:51.509 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:51.509 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:51.509 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:51.512 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:51.545 INFO 2458 --- [ocp-schedules-18,bbaba1e9c6edeed2,1b743ba310de18ba] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:08:51.549 INFO 2458 --- [ocp-schedules-10,fdfbe79b2e716ea0,e15abdab4da9d49e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:08:51.550 INFO 2458 --- [ocp-schedules-18,bbaba1e9c6edeed2,1b743ba310de18ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:51.554 INFO 2458 --- [ocp-schedules-14,f2a1480275e9197a,7aad984f4210fdfe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:08:51.554 INFO 2458 --- [ocp-schedules-10,fdfbe79b2e716ea0,e15abdab4da9d49e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:51.554 INFO 2458 --- [ocp-schedules-10,fdfbe79b2e716ea0,e15abdab4da9d49e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:08:51.555 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:08:51.557 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:08:51.556807604+08:00 2025-10-27 11:08:51.557 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:08:51.558 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:08:51.557+08:00 2025-10-27 11:08:51.559 INFO 2458 --- [ocp-schedules-14,f2a1480275e9197a,7aad984f4210fdfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:51.560 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:08:51.558623791+08:00 2025-10-27 11:08:51.560 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:08:51.560+08:00 2025-10-27 11:08:51.563 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:08:51.565 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:08:51.565491957+08:00 2025-10-27 11:08:51.566 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:08:51.565+08:00 2025-10-27 11:08:51.569 INFO 2458 --- [ocp-schedules-27,59cdb0879d577a35,4651ec6644338148] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:08:51.569 INFO 2458 --- [ocp-schedules-28,29c65937249c1e1d,6c13a9bbb27fb1eb] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:08:51.570 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:08:51.570 INFO 2458 --- [ocp-schedules-15,d1c523409cc100b0,14f21217c1ac1c7b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:08:51.570 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:08:51.572 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:08:51.572264310+08:00 2025-10-27 11:08:51.573 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:08:51.572+08:00 2025-10-27 11:08:51.573 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:08:51.573 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:08:51.573 INFO 2458 --- [host-check-agent-status2,71c7eda4fcce167d,5725e9660458e78d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:08:51.585 INFO 2458 --- [ocp-schedules-27,59cdb0879d577a35,4651ec6644338148] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:08:51.585 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:08:51.560+08:00 and status AVAILABLE 2025-10-27 11:08:51.585 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:08:51.565+08:00 and status AVAILABLE 2025-10-27 11:08:51.586 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:08:51.587 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:08:51.557+08:00 and status AVAILABLE 2025-10-27 11:08:51.588 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:08:51.588 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:08:51.587243293+08:00 2025-10-27 11:08:51.588 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:08:51.588+08:00 2025-10-27 11:08:51.589 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:08:51.589 INFO 2458 --- [ocp-schedules-15,d1c523409cc100b0,14f21217c1ac1c7b] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:08:51.591 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status1,71c7eda4fcce167d,6a314d30a149fd45] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:08:51.592 INFO 2458 --- [host-check-agent-status5,71c7eda4fcce167d,cb98412c19119726] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:08:51.593 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:08:51.572+08:00 and status AVAILABLE 2025-10-27 11:08:51.595 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:08:51.596 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:08:51.598 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:08:51.599 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:08:51.602 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:08:51.602 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:08:51.602 INFO 2458 --- [host-check-agent-status3,71c7eda4fcce167d,dcb9b276dbde6320] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:08:51.603 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:08:51.588+08:00 and status AVAILABLE 2025-10-27 11:08:51.604 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:08:51.606 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:08:51.606 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:08:51.606 INFO 2458 --- [host-check-agent-status4,71c7eda4fcce167d,3480d1cc746e9439] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:08:51.607 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:08:51.646 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:51.648 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.649 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:51.650 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:08:51.653 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:51.654 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:08:51.655 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:08:51.658 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:08:51.658 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:51.658 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:08:51.659 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:08:51.662 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:08:51.663 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:51.674 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.674 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.675 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.675 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.675 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.676 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:51.680 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:08:51.681 INFO 2458 --- [ocp-schedules-18,bbaba1e9c6edeed2,1b743ba310de18ba] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:08:51.702 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.703 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:08:51.715581+08:00, lastProcessTime=2025-10-27T11:08:41.714036+08:00 2025-10-27 11:08:51.732 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:51.734 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:51.734 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:51.735 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:51.739 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:08:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.741 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:51.751 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:08:51.753 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:08:51.754 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:08:51.754 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:08:51.763 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:08:51.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:08:51.794 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:08:51.795 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:08:51.799 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:08:51.800 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:08:51.800 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:08:51.839 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:51.839 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.839 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.840 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.840 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.840 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:08:51.858 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:51.865 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:51.871 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:51.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:51.874 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.874 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:51.875 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:51.876 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:08:51.879 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:08:51.880 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:08:51.882 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.885 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:08:51.885 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:51.892 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:08:51.893 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:08:51.921 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:08:51.921 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:08:51.922 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:08:51.923 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:51.924 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:51.933 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:08:51.934 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:08:51.952 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.954 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:08:51.955 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:51.955 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:51.956 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:51.956 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:08:51.959 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:08:51.959 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:51.962 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:51.966 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:51.975 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:51.976 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:51.978 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:08:51.979 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:08:51.982 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:08:51.991 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:08:51.995 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:08:52.009 INFO 2458 --- [host-check-host-status5,aa8efeed78a2b9a8,e1d055c67ee95667] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:08:52.010 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.011 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:52.011 INFO 2458 --- [host-check-host-status4,aa8efeed78a2b9a8,34746811acc44c31] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:08:52.011 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:52.012 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.013 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:52.014 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:08:52.014 INFO 2458 --- [host-check-host-status2,aa8efeed78a2b9a8,e377d03d4144dd4f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:08:52.023 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:08:52.032 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:08:51.715581+08:00 2025-10-27 11:08:52.034 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:08:52.038 INFO 2458 --- [host-check-host-status3,aa8efeed78a2b9a8,9ce6bc01e4bb4340] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:08:52.045 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:08:52.048 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:08:52.053 INFO 2458 --- [host-check-host-status1,aa8efeed78a2b9a8,6e2460f9832d9183] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:08:52.058 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.059 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:52.095 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:08:52.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:08:52.139 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:08:52.139 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.140 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:08:52.146 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.147 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:08:52.163 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:08:52.169 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.169 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:08:52.185 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:52.190 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.191 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:08:52.210 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:08:52.217 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:08:52.217 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.250 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:08:52.256 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:08:52.267 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:08:52.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:08:52.281 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:08:52.336 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:08:52.385 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:08:52.385 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.385 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.386 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.386 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:08:52.388 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.388 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.388 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.389 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:08:52.407 INFO 2458 --- [host-check-clockdiff2,27d8a95d67640026,cc436c442df20a2c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534532 -1 -1 , error= 2025-10-27 11:08:52.408 INFO 2458 --- [host-check-clockdiff3,27d8a95d67640026,8e7bcb0ff021aa99] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534532 0 0 , error= 2025-10-27 11:08:52.437 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:08:52.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:52.479 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:52.480 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:08:52.509 WARN 2458 --- [ocp-async-6,f10439dc81643b5c,5e575b37b99285f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:52.509 INFO 2458 --- [ocp-async-8,fdca1d33642aaf6b,cd11d7f6ade485d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:52.525 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:08:52.525 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:08:52.525 INFO 2458 --- [ocp-schedules-14,f2a1480275e9197a,7aad984f4210fdfe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:08:53.001 INFO 2458 --- [ocp-schedules-9,79838dc340cf2245,e7ec01edb7117a1b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:08:53.062 WARN 2458 --- [scanner-112137964-1,79838dc340cf2245,bcb1cac559b6fb3c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:08:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:08:53.670 INFO 2458 --- [ocp-schedules-7,35adb5b8748fa979,ae953d16cc7bb3a7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:08:53.674 INFO 2458 --- [ocp-schedules-7,35adb5b8748fa979,ae953d16cc7bb3a7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:08:53.707 INFO 2458 --- [prometheus-minute-schedule-1,91e5ba98ffd3e645,02748864ce3b2f97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:53.714 INFO 2458 --- [prometheus-minute-schedule-1,91e5ba98ffd3e645,02748864ce3b2f97] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:54.013 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:54.014 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:54.466 INFO 2458 --- [ocp-schedules-15,c020ff504111eacc,f0180376e4f1e75a] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:08:54.509 WARN 2458 --- [ocp-async-2,3775aef0a9473297,f1c624ba93563734] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:54.509 INFO 2458 --- [ocp-async-9,2346aba1e2c4d20c,adc17aac28f17a01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:54.580 INFO 2458 --- [ocp-schedules-25,986b61923820901e,e3d5218f21f8d655] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-10-27 11:08:54.580 INFO 2458 --- [ocp-schedules-25,986b61923820901e,e3d5218f21f8d655] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-10-27 11:08:55.895 INFO 2458 --- [ocp-schedules-11,3dc63639286df23e,f910d83498f89691] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:08:56.509 WARN 2458 --- [ocp-async-10,20489984b377d9f8,1d1da6cf0d221c22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:56.509 INFO 2458 --- [ocp-async-10,df6d30b9eb9bc319,6caefd1de5621968] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:56.707 INFO 2458 --- [prometheus-minute-schedule-1,366c1aa05397f108,295d77f342d32e5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:56.714 INFO 2458 --- [prometheus-minute-schedule-1,366c1aa05397f108,295d77f342d32e5f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:08:57.014 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:08:57.015 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:08:58.229 INFO 2458 --- [ocp-schedules-21,1578eeccab4bffb5,66674621692c4b7c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:08:58.236 INFO 2458 --- [ocp-schedules-21,1578eeccab4bffb5,66674621692c4b7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:08:58.319 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:08:58.319 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.324 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:08:58.336 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.336 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.363 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.364 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.377 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:08:58.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.378 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.381 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:08:58.400 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.401 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.403 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.403 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.404 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.404 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.404 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.405 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.430 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.430 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.435 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.436 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.454 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.455 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.462 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.462 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:08:58.465 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.465 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.465 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.466 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.486 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.487 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.488 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.489 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.490 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.490 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.491 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.492 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.493 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.493 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.494 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.495 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.495 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.495 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.495 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.495 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:08:58.496 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.502 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:08:58.509 WARN 2458 --- [ocp-async-1,1ed87df88923889c,555e94cea2fb23ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:08:58.509 INFO 2458 --- [ocp-async-1,de879361dfcd3e77,1aca47a036c249cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:08:58.510 INFO 2458 --- [ocp-async-8,cb700855db80c1fc,e2717b8d82e4cc13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:08:58.529 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.529 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.529 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.530 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:08:58.535 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.535 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.535 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.535 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:08:58.559 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:58.560 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.560 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:08:58.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.569 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:08:58.571 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.571 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:08:58.571 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.571 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:08:58.574 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:08:58.574 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:08:58.598 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.598 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.598 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.599 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:08:58.601 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.601 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.601 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:08:58.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.630 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:08:58.656 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.656 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.656 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.657 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:08:58.701 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.701 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.702 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.702 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.703 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.703 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.703 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.704 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.704 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.704 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.704 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.705 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.706 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.706 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.706 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.707 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.707 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.708 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.708 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.708 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.708 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.709 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.709 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.709 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.710 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.710 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.711 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.711 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.711 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.712 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.712 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.712 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.712 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.713 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.713 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.713 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.714 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.714 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:08:58.715 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.715 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.715 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.716 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:08:58.716 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.716 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.716 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.717 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:08:58.718 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:08:58.720 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:08:58.797 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:08:58.798 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:08:58.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:08:58.827 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:08:58.827 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:08:58.827 INFO 2458 --- [ocp-schedules-21,1578eeccab4bffb5,66674621692c4b7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:08:59.196 INFO 2458 --- [ocp-schedules-27,7805a2ac79e008f9,1cdbda52f2444a38] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:08:59.707 INFO 2458 --- [prometheus-minute-schedule-1,6736372a9bc0d36c,84f54a0f1977a91d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534540, interval=60 2025-10-27 11:08:59.714 INFO 2458 --- [prometheus-minute-schedule-1,6736372a9bc0d36c,84f54a0f1977a91d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:00.015 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:00.017 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:00.509 INFO 2458 --- [ocp-async-2,6dc0bbb639b228ef,3660c87d2114fd1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:00.509 WARN 2458 --- [ocp-async-4,0952ad4d7bbf2491,4c442a405ca9388e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:01.714477+08:00, lastProcessTime=2025-10-27T11:08:51.715581+08:00 2025-10-27 11:09:01.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:01.714477+08:00 2025-10-27 11:09:02.238 INFO 2458 --- [ocp-schedules-25,6ff042bb687f02a4,a967edb3675ab640] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:09:02.246 INFO 2458 --- [ocp-schedules-25,6ff042bb687f02a4,a967edb3675ab640] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:02.281 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:02.282 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:09:02.303 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:02.303 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:02.303 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:02.303 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:02.303 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:02.304 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:09:02.323 INFO 2458 --- [ocp-schedules-25,6ff042bb687f02a4,a967edb3675ab640] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:09:02.510 WARN 2458 --- [ocp-async-5,2b5bd597017b40a6,e8f3ce382cb24c7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:02.510 INFO 2458 --- [ocp-async-5,59dac402a1c7bbd3,e33b2d122009016a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:02.707 INFO 2458 --- [prometheus-minute-schedule-1,370b6323c6abbc2b,a5b94ddb6396b07d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:02.726 INFO 2458 --- [prometheus-minute-schedule-1,370b6323c6abbc2b,a5b94ddb6396b07d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:03.017 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:03.018 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:03.142 INFO 2458 --- [ocp-schedules-10,10d1d2e8f0626c8c,91f2b6d7a7ac5348] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:03.212 WARN 2458 --- [scanner-112138149-1,10d1d2e8f0626c8c,ca9acbb1446a1729] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:03.661 INFO 2458 --- [alarm-detect-4,10d1d2e8f0626c8c,e71b0f1b316475ab] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534510 ~ 1761534533 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534525, value=62.518)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534525, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534525=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:09:03.681 INFO 2458 --- [ocp-schedules-28,a53ca16b3428004f,c78b16a22a7b9126] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:03.684 INFO 2458 --- [ocp-schedules-28,a53ca16b3428004f,c78b16a22a7b9126] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:04.510 INFO 2458 --- [ocp-async-6,17f7de98e630b5c6,b85d3e760ea10c98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:04.510 WARN 2458 --- [ocp-async-8,c8f37a20b9170a1b,083e6559232fc132] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:05.707 INFO 2458 --- [prometheus-minute-schedule-1,de90caa218065a08,159607ca46b49c5f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:05.715 INFO 2458 --- [prometheus-minute-schedule-1,de90caa218065a08,159607ca46b49c5f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:05.902 INFO 2458 --- [ocp-schedules-6,bc63321b83aa8906,a133b335cb18c01a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:06.018 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:06.019 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:06.510 INFO 2458 --- [ocp-async-1,76b9fef86045afc8,547e43238bed3250] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:06.510 WARN 2458 --- [ocp-async-4,bbc0b3bb0fabb3b9,7db70665521f73a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:06.774 INFO 2458 --- [ocp-schedules-30,b7e4796fc40aafb7,6cd0dd8a14d76899] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:09:06.783 INFO 2458 --- [ocp-schedules-30,b7e4796fc40aafb7,6cd0dd8a14d76899] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:06.783 INFO 2458 --- [ocp-schedules-30,b7e4796fc40aafb7,6cd0dd8a14d76899] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:08.510 INFO 2458 --- [ocp-async-2,070a401809c54047,6d1a68a58c149742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:08.510 WARN 2458 --- [ocp-async-10,030d1d984fc062f3,f14ab1d72d9e8274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:08.510 INFO 2458 --- [ocp-async-5,db27685a2ccbe3cd,fb14977f033fdd58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:08.707 INFO 2458 --- [prometheus-minute-schedule-1,72996a1f1eb7c477,feea509701dc9e19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:08.715 INFO 2458 --- [prometheus-minute-schedule-1,72996a1f1eb7c477,feea509701dc9e19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:09.019 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:09.019 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:10.012 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:09:10.013 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:09:10.050 INFO 2458 --- [Check obproxy1,f53f67c2cf46eafe,61108d6ee8032ee4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:09:10.050 INFO 2458 --- [Check obproxy1,f53f67c2cf46eafe,61108d6ee8032ee4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:09:10.054 INFO 2458 --- [Collect obproxy1,2668ed9715c02c94,5579203bfca11941] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:09:10.054 INFO 2458 --- [Collect obproxy1,2668ed9715c02c94,5579203bfca11941] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:09:10.085 INFO 2458 --- [Check obproxy1,f53f67c2cf46eafe,61108d6ee8032ee4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:09:10.088 INFO 2458 --- [Collect obproxy1,2668ed9715c02c94,5579203bfca11941] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:09:10.088 INFO 2458 --- [Check obproxy1,f53f67c2cf46eafe,61108d6ee8032ee4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:09:10.088 INFO 2458 --- [Check obproxy1,f53f67c2cf46eafe,61108d6ee8032ee4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:09:10.092 INFO 2458 --- [Collect obproxy1,2668ed9715c02c94,5579203bfca11941] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:09:10.095 INFO 2458 --- [Collect obproxy1,2668ed9715c02c94,5579203bfca11941] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:09:10.186 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:09:10.510 INFO 2458 --- [ocp-async-2,b1f9467bc337d525,ece50d2122817819] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:10.510 WARN 2458 --- [ocp-async-6,f82f05d8cbcd24c3,ab9425e0922d5d4f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:11.707 INFO 2458 --- [prometheus-minute-schedule-1,ebd0f9389e387794,f9c42972a39c95f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:11.714 INFO 2458 --- [prometheus-minute-schedule-1,ebd0f9389e387794,f9c42972a39c95f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:11.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:11.713656+08:00, lastProcessTime=2025-10-27T11:09:01.714477+08:00 2025-10-27 11:09:11.924 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:11.713656+08:00 2025-10-27 11:09:12.019 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:12.021 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:12.510 INFO 2458 --- [ocp-async-4,b59b5973fff6bd33,1d6ba34696406b99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:12.511 WARN 2458 --- [ocp-async-1,82de82b2a374cf17,183e8e3d4654fb0f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:13.685 INFO 2458 --- [ocp-schedules-7,85003e94f3888447,7fa4fd047366420e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:13.700 INFO 2458 --- [ocp-schedules-6,def6151815235fde,372bb49b0322279f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:13.704 INFO 2458 --- [ocp-schedules-6,def6151815235fde,372bb49b0322279f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:13.725 WARN 2458 --- [scanner-112139932-1,85003e94f3888447,76c84bf70db22b42] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:13.798 INFO 2458 --- [alarm-detect-10,85003e94f3888447,fa4d82a2fb37ebce] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534525 ~ 1761534543 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534540, value=51.867)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534540, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534540=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:09:14.018 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534480, nodeCount=415, affectRows=415 2025-10-27 11:09:14.031 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534480, nodeCount=406, affectRows=406 2025-10-27 11:09:14.417 INFO 2458 --- [ocp-schedules-24,32115bdd81459a5f,b3036e19ca76ad2a] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:09:14.417+08:00, clusters=[1, 2] 2025-10-27 11:09:14.419 INFO 2458 --- [ocp-schedules-24,32115bdd81459a5f,b3036e19ca76ad2a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:09:14.421 INFO 2458 --- [ocp-schedules-24,32115bdd81459a5f,b3036e19ca76ad2a] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:09:14.474 INFO 2458 --- [ocp-schedules-21,690aa4052b43b051,9fab4ee173377a4e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:09:14.474 INFO 2458 --- [ocp-schedules-21,690aa4052b43b051,9fab4ee173377a4e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:09:14.511 INFO 2458 --- [ocp-async-7,88ca9ab42a9b1b51,f6596ebc761bb62a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:14.511 WARN 2458 --- [ocp-async-8,af1ccc93b7c2d0c2,d38cc9f903fb78bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:14.526 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:09:14.528 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:09:14.528 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:09:14.529 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:09:14.529 INFO 2458 --- [Collect all tenants major compactions6,690aa4052b43b051,2a50568c8eeb415b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:09:14.530 INFO 2458 --- [Collect all tenants major compactions5,690aa4052b43b051,5d6845cde74a6223] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:09:14.567 INFO 2458 --- [Collect all tenants major compactions5,690aa4052b43b051,5d6845cde74a6223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.568 INFO 2458 --- [Collect all tenants major compactions6,690aa4052b43b051,2a50568c8eeb415b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.568 INFO 2458 --- [Collect all tenants major compactions5,690aa4052b43b051,5d6845cde74a6223] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:09:14.569 INFO 2458 --- [Collect all tenants major compactions6,690aa4052b43b051,2a50568c8eeb415b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:09:14.579 INFO 2458 --- [Collect all tenants major compactions6,690aa4052b43b051,2a50568c8eeb415b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:09:14.581 INFO 2458 --- [Collect all tenants major compactions5,690aa4052b43b051,5d6845cde74a6223] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:09:14.585 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.585 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.586 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.586 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.586 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.587 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:09:14.588 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.588 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.588 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.588 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.588 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.589 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:09:14.590 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.590 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.590 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:14.590 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:14.590 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:14.592 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:09:14.592 INFO 2458 --- [Collect all tenants major compactions3,690aa4052b43b051,91143fe4847b7bc5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:09:14.592 INFO 2458 --- [Collect all tenants major compactions1,690aa4052b43b051,4fa1a7961052ea2a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:09:14.593 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:09:14.595 INFO 2458 --- [Collect all tenants major compactions4,690aa4052b43b051,84a17f7b60b3e5a2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:09:14.596 INFO 2458 --- [Collect all tenants major compactions2,690aa4052b43b051,90118934a8258f79] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:09:14.597 INFO 2458 --- [ocp-schedules-21,690aa4052b43b051,9fab4ee173377a4e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:09:14.707 INFO 2458 --- [prometheus-minute-schedule-1,7ae1568ac37bbf5c,5b733b602358ce90] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:14.723 INFO 2458 --- [prometheus-minute-schedule-1,7ae1568ac37bbf5c,5b733b602358ce90] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:15.021 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:15.022 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:15.484 INFO 2458 --- [ocp-schedules-29,84500196afdb2925,0258ddb3a0dc7826] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:09:15.484 INFO 2458 --- [ocp-schedules-29,84500196afdb2925,0258ddb3a0dc7826] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:09:15.492 INFO 2458 --- [ocp-schedules-29,84500196afdb2925,0258ddb3a0dc7826] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:09:15.492 INFO 2458 --- [ocp-schedules-29,84500196afdb2925,0258ddb3a0dc7826] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:09:15.912 INFO 2458 --- [ocp-schedules-15,28aeeeee361bad7a,cd925bcb137a8d89] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:16.511 INFO 2458 --- [ocp-async-6,0212800edef29946,339f393842ae6a28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:16.511 WARN 2458 --- [ocp-async-3,e895c55125ad6151,e770cbb21146a796] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:17.707 INFO 2458 --- [prometheus-minute-schedule-1,8fcb57c33cd904c1,42e694786223bc0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:17.714 INFO 2458 --- [prometheus-minute-schedule-1,8fcb57c33cd904c1,42e694786223bc0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:17.745 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835370602979598336-1761534420' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835370602979598336-1761534420' for key 'PRIMARY' 2025-10-27 11:09:17.760 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134521640) Duplicate entry '835014129044889601-1761534420' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134521640) Duplicate entry '835014129044889601-1761534420' for key 'PRIMARY' 2025-10-27 11:09:17.950 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=1000, affectRows=1000 2025-10-27 11:09:17.950 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=1000, affectRows=1000 2025-10-27 11:09:17.950 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=77, affectRows=77 2025-10-27 11:09:17.951 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=75, validCount=75 2025-10-27 11:09:17.955 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=1000, affectRows=1000 2025-10-27 11:09:17.955 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=996, validCount=992 2025-10-27 11:09:17.955 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=994, validCount=992 2025-10-27 11:09:17.959 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=999, validCount=995 2025-10-27 11:09:18.022 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:18.023 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:18.511 INFO 2458 --- [ocp-async-2,9998dc251de1ce49,05480fb8aed65846] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:18.511 WARN 2458 --- [ocp-async-9,a66f90d07ddadff9,de856be3f49e8383] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:18.511 INFO 2458 --- [ocp-async-4,44e70e7ca2440e07,1c3f3b20a0786b0b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:19.287 INFO 2458 --- [ocp-schedules-7,2baf587a799278ea,6d4503657cd70527] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:09:20.511 WARN 2458 --- [ocp-async-10,e976e28b2718bde5,5ee59caf7858c3dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:20.511 INFO 2458 --- [ocp-async-10,8f203b0dcc41313c,d3dbb8b66e481286] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:20.707 INFO 2458 --- [prometheus-minute-schedule-1,b444fd7db77b9a39,18a2e49f29f9b12c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:20.715 INFO 2458 --- [prometheus-minute-schedule-1,b444fd7db77b9a39,18a2e49f29f9b12c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:21.023 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:21.023 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:21.412 INFO 2458 --- [ocp-schedules-11,2dcbc82b4fcd4963,10dc589faeb80d1a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:21.412 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:09:21.413 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:09:21.413 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:09:21.413 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:09:21.413 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:09:21.417 INFO 2458 --- [ocp-schedules-16,4a542fca45a80f7c,c407541c9a74d5ee] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:21.418 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:09:21.420 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:09:21.420 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:09:21.420 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:09:21.422 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:09:21.429 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:21.429 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:21.429 INFO 2458 --- [ocp-schedules-29,29c4d54f1042b596,0d480d9495889990] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:21.430 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:21.432 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:21.432 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:21.434 INFO 2458 --- [host-check-clockdiff1,29c4d54f1042b596,4fa7886cc5ac2119] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:09:21.435 INFO 2458 --- [host-check-clockdiff2,29c4d54f1042b596,7ba52be1c6416907] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:09:21.435 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:21.436 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:21.436 INFO 2458 --- [host-check-clockdiff3,29c4d54f1042b596,d1926125a6efa950] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:09:21.436 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:21.436 INFO 2458 --- [host-check-clockdiff4,29c4d54f1042b596,57db12fd440cd484] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:09:21.437 INFO 2458 --- [host-check-clockdiff5,29c4d54f1042b596,1e67848c2571802c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:09:21.442 INFO 2458 --- [ocp-schedules-14,a2ddd4687c85ec9f,857a799f2e19b21e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:09:21.442 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:21.452 INFO 2458 --- [host-check-clockdiff1,29c4d54f1042b596,4fa7886cc5ac2119] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534561 0 0 , error= 2025-10-27 11:09:21.452 INFO 2458 --- [host-check-clockdiff2,29c4d54f1042b596,7ba52be1c6416907] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534561 -1 -1 , error= 2025-10-27 11:09:21.454 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:21.455 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:21.457 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:21.457 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:21.461 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:21.461 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:21.461 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:21.463 INFO 2458 --- [host-check-clockdiff3,29c4d54f1042b596,d1926125a6efa950] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534561 0 0 , error= 2025-10-27 11:09:21.465 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:21.465 INFO 2458 --- [host-check-clockdiff5,29c4d54f1042b596,1e67848c2571802c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534561 0 0 , error= 2025-10-27 11:09:21.465 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:21.465 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:21.500 INFO 2458 --- [host-check-clockdiff4,29c4d54f1042b596,57db12fd440cd484] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534561 -1 -1 , error= 2025-10-27 11:09:21.502 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:21.545 INFO 2458 --- [ocp-schedules-22,0d90f7282587f15c,9c802c09e94d4bb8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:09:21.546 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:09:21.548 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:09:21.548 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:09:21.548 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:09:21.548842791+08:00 2025-10-27 11:09:21.549 INFO 2458 --- [ocp-schedules-9,ef91ad854f93145c,aacb5451dc06ac2b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:09:21.549 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:09:21.549+08:00 2025-10-27 11:09:21.550 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:09:21.549745220+08:00 2025-10-27 11:09:21.551 INFO 2458 --- [ocp-schedules-22,0d90f7282587f15c,9c802c09e94d4bb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:21.551 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:09:21.550+08:00 2025-10-27 11:09:21.552 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:09:21.553 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:09:21.553 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:09:21.553 INFO 2458 --- [host-check-agent-status1,4a542fca45a80f7c,e662e20d9a98e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:09:21.554 INFO 2458 --- [ocp-schedules-12,ab0b4d765452336f,5eda820ffb8e1ae8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:09:21.554 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:09:21.555 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:09:21.555232468+08:00 2025-10-27 11:09:21.555 INFO 2458 --- [ocp-schedules-9,ef91ad854f93145c,aacb5451dc06ac2b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:21.556 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:09:21.555+08:00 2025-10-27 11:09:21.556 INFO 2458 --- [ocp-schedules-9,ef91ad854f93145c,aacb5451dc06ac2b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:09:21.556 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:09:21.556 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:09:21.556 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:09:21.557 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:09:21.557 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:09:21.557 INFO 2458 --- [host-check-agent-status2,4a542fca45a80f7c,7e5fb7d20d3d0d6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:09:21.557 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:09:21.556168871+08:00 2025-10-27 11:09:21.558 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:09:21.557+08:00 2025-10-27 11:09:21.558 INFO 2458 --- [ocp-schedules-12,ab0b4d765452336f,5eda820ffb8e1ae8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:21.559 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:09:21.558151536+08:00 2025-10-27 11:09:21.559 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:09:21.559+08:00 2025-10-27 11:09:21.560 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:09:21.560 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:09:21.560 INFO 2458 --- [host-check-agent-status4,4a542fca45a80f7c,32334d6936bba2ef] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:09:21.561 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:09:21.550+08:00 and status AVAILABLE 2025-10-27 11:09:21.563 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:09:21.549+08:00 and status AVAILABLE 2025-10-27 11:09:21.565 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:09:21.555+08:00 and status AVAILABLE 2025-10-27 11:09:21.566 INFO 2458 --- [ocp-schedules-19,0d63d56ce878689a,ab3a96d21602237f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:09:21.566 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:09:21.566 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:09:21.557+08:00 and status AVAILABLE 2025-10-27 11:09:21.566 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:09:21.568 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:09:21.568 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:09:21.559+08:00 and status AVAILABLE 2025-10-27 11:09:21.569 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:09:21.569 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:09:21.569 INFO 2458 --- [host-check-agent-status5,4a542fca45a80f7c,72c6bd850c108049] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:09:21.569 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:09:21.571 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:09:21.572 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:09:21.579 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:09:21.582 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:09:21.582 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:09:21.582 INFO 2458 --- [host-check-agent-status3,4a542fca45a80f7c,3e7d0b8547d6ec00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:09:21.596 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.596 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:21.598 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:09:21.604 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:21.605 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:21.607 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:21.608 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:21.608 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:21.609 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:21.610 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:21.611 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:21.612 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:21.614 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:21.632 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:09:21.632 INFO 2458 --- [ocp-schedules-22,0d90f7282587f15c,9c802c09e94d4bb8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:09:21.643 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.644 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:21.671 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:21.675 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:21.677 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:21.679 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:21.681 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:21.690 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:09:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.692 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:21.700 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:09:21.701 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:09:21.701 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:09:21.703 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:21.703 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:09:21.703 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.704 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:09:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:21.715562+08:00, lastProcessTime=2025-10-27T11:09:11.713656+08:00 2025-10-27 11:09:21.724 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:09:21.735 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:09:21.740 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:09:21.740 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:09:21.744 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:09:21.757 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:21.766 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:21.769 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:21.770 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.771 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.772 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:09:21.773 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:09:21.773 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:09:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:21.775 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:21.781 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:21.784 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:09:21.793 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:09:21.801 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:09:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:21.808 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:09:21.809 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:09:21.831 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:09:21.833 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:21.835 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:09:21.836 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:09:21.840 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:21.843 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:09:21.855 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:09:21.861 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:21.862 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.863 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:21.864 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:09:21.865 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:21.867 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.868 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:21.869 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:21.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:09:21.873 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:21.875 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:21.877 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:09:21.884 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:09:21.888 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:09:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:21.889 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:09:21.890 INFO 2458 --- [host-check-host-status4,2dcbc82b4fcd4963,fea90df01f70bd18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:09:21.891 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:21.904 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:09:21.921 INFO 2458 --- [host-check-host-status5,2dcbc82b4fcd4963,cf266a8233d55883] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:09:21.922 INFO 2458 --- [host-check-host-status3,2dcbc82b4fcd4963,48d48bef77efaba8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:09:21.924 INFO 2458 --- [host-check-host-status2,2dcbc82b4fcd4963,ffa6c550b6f2db40] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:09:21.925 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:21.925 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:21.935 INFO 2458 --- [host-check-host-status1,2dcbc82b4fcd4963,51a265850ffb35f2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:09:21.955 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:09:21.966 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:21.966 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:21.978 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:21.715562+08:00 2025-10-27 11:09:21.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:21.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:21.999 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:09:21.999 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:09:22.002 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.003 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:09:22.018 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:09:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:22.043 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:22.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:09:22.067 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:09:22.067 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:09:22.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:22.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:22.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:22.145 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:22.145 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.145 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:22.150 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:09:22.161 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:09:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:22.175 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:22.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:09:22.280 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:09:22.280 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.280 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.281 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.281 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:09:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.284 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:09:22.326 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:09:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:22.366 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:09:22.408 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:09:22.408 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:09:22.408 INFO 2458 --- [ocp-schedules-12,ab0b4d765452336f,5eda820ffb8e1ae8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:09:22.511 WARN 2458 --- [ocp-async-5,82719a592dc44ebb,dceda66dd1595f3b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:22.511 INFO 2458 --- [ocp-async-6,1318bfdd90fa3d0f,1ecca65cf738f888] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:22.673 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=1000, affectRows=1000 2025-10-27 11:09:22.678 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=1000, validCount=989 2025-10-27 11:09:22.768 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534480, nodeCount=1000, affectRows=1000 2025-10-27 11:09:22.773 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534480, nodeCount=996, validCount=990 2025-10-27 11:09:23.707 INFO 2458 --- [prometheus-minute-schedule-1,bfbca796a9f7c065,b486319c78cc74a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:23.712 INFO 2458 --- [ocp-schedules-10,796823d542e14bd3,be3303483912a081] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:23.714 INFO 2458 --- [prometheus-minute-schedule-1,bfbca796a9f7c065,b486319c78cc74a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:23.723 INFO 2458 --- [ocp-schedules-10,796823d542e14bd3,be3303483912a081] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:23.805 INFO 2458 --- [ocp-schedules-29,0004312d1e6aca68,d33ad49b6437e3f8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:23.836 WARN 2458 --- [scanner-112140112-1,0004312d1e6aca68,d92e483e2013bbae] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:24.024 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:24.024 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:24.511 WARN 2458 --- [ocp-async-9,5f80b53a69e01c4b,00a1ee8a249122a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:24.511 INFO 2458 --- [ocp-async-1,4730539421151bee,8793502e4b7a3ff0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:25.919 INFO 2458 --- [ocp-schedules-3,83e499a1725e1127,63618e538f105dae] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:26.511 WARN 2458 --- [ocp-async-2,a376954db5652e7d,74733c66590897b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:26.511 INFO 2458 --- [ocp-async-7,7d0c52a4614920e2,bdef973308571153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:26.707 INFO 2458 --- [prometheus-minute-schedule-1,acef5a2649071252,2bbac493ef26f06d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:26.714 INFO 2458 --- [prometheus-minute-schedule-1,acef5a2649071252,2bbac493ef26f06d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:26.786 INFO 2458 --- [ocp-schedules-2,e312c895525e5bcb,68ef23a29b051dfd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:09:26.794 INFO 2458 --- [ocp-schedules-2,e312c895525e5bcb,68ef23a29b051dfd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:26.795 INFO 2458 --- [ocp-schedules-2,e312c895525e5bcb,68ef23a29b051dfd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:27.024 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:27.025 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:28.511 INFO 2458 --- [ocp-async-7,c66218743edad21d,d081cd997e197824] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:28.511 WARN 2458 --- [ocp-async-5,f5a61dede935f05e,69c2e420e0f32e9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:28.511 INFO 2458 --- [ocp-async-5,e2c803d1aa32c75a,3f8286bc142243d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:28.862 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:09:28.862 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:09:28.864 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:09:28.864 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:09:28.865 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:09:28.865 INFO 2458 --- [ocp-schedules-28,1ba7fa2c624e3161,0e028a62d2bdd3a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:09:29.324 INFO 2458 --- [ocp-schedules-27,2c9da0afb0d0ba3f,353cff5be7d12913] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:09:29.328 INFO 2458 --- [ocp-schedules-27,2c9da0afb0d0ba3f,353cff5be7d12913] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:09:29.328 INFO 2458 --- [ocp-schedules-27,2c9da0afb0d0ba3f,353cff5be7d12913] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:09:29.707 INFO 2458 --- [prometheus-minute-schedule-1,9d9c18acad70697f,2119b61a7dc35b00] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:29.715 INFO 2458 --- [prometheus-minute-schedule-1,9d9c18acad70697f,2119b61a7dc35b00] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:30.025 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:30.026 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:30.512 INFO 2458 --- [ocp-async-6,451a17fbb1a82658,7fc6b09abe692386] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:30.512 WARN 2458 --- [ocp-async-4,ca4f4b9ac9953c76,cbf01ed273443287] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:31.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:31.715147+08:00, lastProcessTime=2025-10-27T11:09:21.715562+08:00 2025-10-27 11:09:31.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:31.715147+08:00 2025-10-27 11:09:32.093 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,3d9ed7c5b1ac7d92,1177777aea3581e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=3d9ed7c5b1ac7d92, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:09:32.093 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,3d9ed7c5b1ac7d92,1177777aea3581e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=3d9ed7c5b1ac7d92, duration=1 ms] 2025-10-27 11:09:32.512 WARN 2458 --- [ocp-async-8,668f05406321be90,d036a2f0a91d70f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:32.512 INFO 2458 --- [ocp-async-1,08900e4885e15ef4,1ca9ffa8c9848d96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:32.707 INFO 2458 --- [prometheus-minute-schedule-1,66a7f773536efafd,88ec7e9589897efd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:32.714 INFO 2458 --- [prometheus-minute-schedule-1,66a7f773536efafd,88ec7e9589897efd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:33.026 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:33.027 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:33.731 INFO 2458 --- [ocp-schedules-27,d10ef020ac0297da,ae5fc7607004d586] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:33.735 INFO 2458 --- [ocp-schedules-27,d10ef020ac0297da,ae5fc7607004d586] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:33.753 INFO 2458 --- [ocp-schedules-2,2ec6838df3e171d5,ab416ab5277bf7ce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:09:33.753 INFO 2458 --- [ocp-schedules-2,2ec6838df3e171d5,ab416ab5277bf7ce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:09:33.981 INFO 2458 --- [ocp-schedules-1,72040daba8e6bf2d,16c4015f33bd0a39] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:33.983 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:09:33.983 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:09:34.006 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:09:34.006 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:09:34.006 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:09:34.008 WARN 2458 --- [scanner-112140347-1,72040daba8e6bf2d,9b4a22f064705f61] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:34.028 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:09:34.029 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:09:34.029 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:09:34.043 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:09:34.043 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:09:34.043 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:09:34.053 INFO 2458 --- [ocp-schedules-8,39bcc0d32a29b9e6,e84999d24ea156fd] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:09:34.109 INFO 2458 --- [alarm-detect-14,72040daba8e6bf2d,f366b094614a8955] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534540 ~ 1761534564 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534555, value=51.712)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534555, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534555=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:09:34.512 WARN 2458 --- [ocp-async-9,932b255cb3cdf8fb,df8605d5df1ef4d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:34.512 INFO 2458 --- [ocp-async-1,56c8ca1a462b271c,ecd1ec94e435b497] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:35.376 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:09:35.376 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:09:35.446 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:35.447 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:35.510 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:35.510 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:35.510 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:35.511 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:09:35.525 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:35.526 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:35.526 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:35.526 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:09:35.599 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:35.600 INFO 2458 --- [ocp-schedules-19,8d1a7c8bc07924ad,f2b67a5b223d9327] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:09:35.707 INFO 2458 --- [prometheus-minute-schedule-1,d9e9c6ac49233025,6be5e1f388c72a6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:35.714 INFO 2458 --- [prometheus-minute-schedule-1,d9e9c6ac49233025,6be5e1f388c72a6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:35.926 INFO 2458 --- [ocp-schedules-4,478e44e0ba6dd124,370af4774fa55f6d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:36.027 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:36.029 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:36.512 WARN 2458 --- [ocp-async-4,5eb4b914a450c921,1eb5de9e8e1a46ed] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:36.512 INFO 2458 --- [ocp-async-5,a6ebb696ced1fdba,5b3bc7212fbdec16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:37.252 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:37.252 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:09:37.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:09:37.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:09:37.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:09:37.252 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:09:37.264 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:37.264 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:37.264 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:37.265 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:37.265 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:37.268 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:37.269 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:37.269 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:37.269 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:37.269 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:37.324 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:09:37.333 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:09:37.336 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:09:37.339 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:09:37.347 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:09:37.350 INFO 2458 --- [ocp-schedules-7,64e9d1771a5b8370,11ebdbd95822f0c7] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:09:37.708 INFO 2458 --- [prometheus-cache-monitor1,a9bd90c02f4fb574,53bb47610a626959] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20309,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:09:37.708 INFO 2458 --- [prometheus-cache-monitor1,a9bd90c02f4fb574,53bb47610a626959] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:09:38.511 INFO 2458 --- [ocp-async-2,facc213a9144f4d6,9546bba95642bbad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:38.512 WARN 2458 --- [ocp-async-6,22c1b1edcf090858,632dcd6602a2dcf8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:38.512 INFO 2458 --- [ocp-async-6,f9c5e31f30b6fcdc,1e9f8d3c74a2d609] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:38.603 INFO 2458 --- [ocp-schedules-7,255f8923adf66c67,98a5de9669043d44] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:09:38.610 INFO 2458 --- [ocp-schedules-7,255f8923adf66c67,98a5de9669043d44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:38.610 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:09:38.610 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:09:38.610 INFO 2458 --- [ocp-schedules-7,255f8923adf66c67,98a5de9669043d44] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:09:38.707 INFO 2458 --- [prometheus-minute-schedule-1,d44a67e82b4121d5,d131b7daf173d943] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:38.714 INFO 2458 --- [prometheus-minute-schedule-1,d44a67e82b4121d5,d131b7daf173d943] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:39.029 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:39.030 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:40.010 INFO 2458 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:09:40.011 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:09:40.050 INFO 2458 --- [Check obproxy1,ea44a489fb438725,563d05bb05da5182] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:09:40.050 INFO 2458 --- [Check obproxy1,ea44a489fb438725,563d05bb05da5182] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:09:40.051 INFO 2458 --- [Collect obproxy1,e172a070b74a5bb7,2121885d83e3802e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:09:40.051 INFO 2458 --- [Collect obproxy1,e172a070b74a5bb7,2121885d83e3802e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:09:40.081 INFO 2458 --- [Check obproxy1,ea44a489fb438725,563d05bb05da5182] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:09:40.083 INFO 2458 --- [Collect obproxy1,e172a070b74a5bb7,2121885d83e3802e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:09:40.084 INFO 2458 --- [Check obproxy1,ea44a489fb438725,563d05bb05da5182] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:09:40.084 INFO 2458 --- [Check obproxy1,ea44a489fb438725,563d05bb05da5182] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:09:40.086 INFO 2458 --- [Collect obproxy1,e172a070b74a5bb7,2121885d83e3802e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:09:40.088 INFO 2458 --- [Collect obproxy1,e172a070b74a5bb7,2121885d83e3802e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:09:40.512 WARN 2458 --- [ocp-async-1,5b1b02f0914298ff,e93a7a32305e1a8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:40.512 INFO 2458 --- [ocp-async-10,3edc3fbb0315f86c,5e7cc3dbd1c96e54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:40.595 INFO 2458 --- [ocp-schedules-2,008129448994b28d,beae5b8e679b453f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:09:40.596 INFO 2458 --- [ocp-schedules-2,008129448994b28d,beae5b8e679b453f] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:09:40.619 INFO 2458 --- [ocp-schedules-2,008129448994b28d,beae5b8e679b453f] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:09:41.707 INFO 2458 --- [prometheus-minute-schedule-1,9c10e601a5f06d92,620555f7c5f458ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:41.714 INFO 2458 --- [prometheus-minute-schedule-1,9c10e601a5f06d92,620555f7c5f458ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:41.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:41.717505+08:00, lastProcessTime=2025-10-27T11:09:31.715147+08:00 2025-10-27 11:09:41.919 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:41.717505+08:00 2025-10-27 11:09:42.030 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:42.031 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:42.512 INFO 2458 --- [ocp-async-7,8c154dcec849c61c,14d2da4c3688255c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:42.512 WARN 2458 --- [ocp-async-5,b6d7fa88fc569a2d,b9f123aa134e2017] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:43.745 INFO 2458 --- [ocp-schedules-8,3d57c61d3c70d2f6,168aefd27f8d94e6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:43.750 INFO 2458 --- [ocp-schedules-8,3d57c61d3c70d2f6,168aefd27f8d94e6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:44.004 INFO 2458 --- [ocp-schedules-3,742b64ce9cbf56f9,74cd1e3c5be7dd72] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:09:44.117 INFO 2458 --- [ocp-schedules-13,15d402c597cf831f,dc07a2f7f39c8bb9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:44.179 WARN 2458 --- [scanner-112140563-1,15d402c597cf831f,e1968abd578f286c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:44.338 INFO 2458 --- [alarm-detect-14,15d402c597cf831f,a6a574e77e02c707] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534555 ~ 1761534574 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534570, value=45.535)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534570, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534570=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:09:44.512 WARN 2458 --- [ocp-async-3,0949e9147c4ab76c,250bbbf8964d106e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:44.513 INFO 2458 --- [ocp-async-8,4a6838ada5cf25d9,9e0c767e5779306e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:44.707 INFO 2458 --- [prometheus-minute-schedule-1,36466065512f489d,2c8b280e8822c727] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:44.715 INFO 2458 --- [prometheus-minute-schedule-1,36466065512f489d,2c8b280e8822c727] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:45.031 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:45.032 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:45.102 INFO 2458 --- [prometheus-second-schedule-1,f82a2bca0720a327,689b446cfcbc1e06] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:09:45.670 INFO 2458 --- [ocp-schedules-5,04f015d6a9b5423c,8a2abd353143f7a0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:09:45.670 INFO 2458 --- [ocp-schedules-5,04f015d6a9b5423c,8a2abd353143f7a0] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:09:45.678 INFO 2458 --- [ocp-schedules-5,04f015d6a9b5423c,8a2abd353143f7a0] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:09:45.678 INFO 2458 --- [ocp-schedules-5,04f015d6a9b5423c,8a2abd353143f7a0] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:09:45.694 INFO 2458 --- [ocp-schedules-28,f28c7d5ddc2d6d9a,4e559445c735ce18] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:09:45.694 INFO 2458 --- [ocp-schedules-28,f28c7d5ddc2d6d9a,4e559445c735ce18] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:09:45.751 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:09:45.751 INFO 2458 --- [pool-877-thread-2,f28c7d5ddc2d6d9a,e5700fc8599dd0ea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:09:45.799 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:45.799 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:45.820 INFO 2458 --- [pool-877-thread-2,f28c7d5ddc2d6d9a,e5700fc8599dd0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:45.848 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:45.848 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:45.848 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:45.852 INFO 2458 --- [pool-877-thread-2,f28c7d5ddc2d6d9a,e5700fc8599dd0ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534285000000, 1761534585000000] 2025-10-27 11:09:45.852 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534285000000, 1761534585000000] 2025-10-27 11:09:45.896 WARN 2458 --- [pool-877-thread-2,f28c7d5ddc2d6d9a,e5700fc8599dd0ea] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:09:45.913 INFO 2458 --- [pool-877-thread-2,f28c7d5ddc2d6d9a,e5700fc8599dd0ea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:09:45.932 INFO 2458 --- [ocp-schedules-19,777ca321e6f285d5,bfe922f5f96ad85f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:46.002 WARN 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:09:46.002 WARN 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:09:46.002 WARN 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:09:46.010 INFO 2458 --- [pool-877-thread-1,f28c7d5ddc2d6d9a,5e1a9a30591ae104] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:09:46.011 INFO 2458 --- [ocp-schedules-28,f28c7d5ddc2d6d9a,4e559445c735ce18] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:09:46.513 WARN 2458 --- [ocp-async-9,b2afa4b557b35df2,ac4f4cb005fa823b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:46.513 INFO 2458 --- [ocp-async-1,470880bf35d19c7b,09111a0026d32f53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:46.797 INFO 2458 --- [ocp-schedules-8,289bb18293666eeb,76261be5cd35b226] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:09:46.805 INFO 2458 --- [ocp-schedules-8,289bb18293666eeb,76261be5cd35b226] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:46.805 INFO 2458 --- [ocp-schedules-8,289bb18293666eeb,76261be5cd35b226] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:09:47.707 INFO 2458 --- [prometheus-minute-schedule-1,67056578b80fda93,5c2bb70e3d161376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:47.715 INFO 2458 --- [prometheus-minute-schedule-1,67056578b80fda93,5c2bb70e3d161376] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:47.916 INFO 2458 --- [ocp-schedules-9,287a994f638da9f7,2dcd36e78bfa2a38] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:09:47.916+08:00, clusters=[1, 2] 2025-10-27 11:09:47.932 INFO 2458 --- [ocp-schedules-9,287a994f638da9f7,2dcd36e78bfa2a38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:09:47.934 INFO 2458 --- [ocp-schedules-9,287a994f638da9f7,2dcd36e78bfa2a38] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:09:48.032 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:48.033 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:48.511 INFO 2458 --- [ocp-async-10,56ca52ef33794627,1f94469919f62840] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:48.513 INFO 2458 --- [ocp-async-5,ccb253706dab2d58,fcd7a7eeb5b2bdd4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:48.513 WARN 2458 --- [ocp-async-4,90abf5f0a7148173,130ccf60f919a30b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:49.060 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,cbbf99de9418d15b,01cd4af288edc47c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=cbbf99de9418d15b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:09:49.060 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,cbbf99de9418d15b,01cd4af288edc47c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=cbbf99de9418d15b, duration=0 ms] 2025-10-27 11:09:50.513 INFO 2458 --- [ocp-async-2,a2bddbeae459ea00,a676466a5f742223] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:50.513 WARN 2458 --- [ocp-async-8,2c281eb261285ff3,4cf18064d9daa219] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:50.707 INFO 2458 --- [prometheus-minute-schedule-1,6e610f1e4e2f0b9b,16fc8102317256e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:50.715 INFO 2458 --- [prometheus-minute-schedule-1,6e610f1e4e2f0b9b,16fc8102317256e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:51.033 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:51.034 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:51.411 INFO 2458 --- [ocp-schedules-1,6f7ec59902de6142,91ab1e83a16a4b47] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:51.411 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:09:51.411 INFO 2458 --- [ocp-schedules-27,da103117873344a7,5c99e64a16ac74f7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:51.411 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:09:51.412 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:09:51.413 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:09:51.414 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:09:51.417 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:09:51.417 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:09:51.417 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:09:51.417 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:09:51.418 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:09:51.427 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:51.427 INFO 2458 --- [ocp-schedules-7,7b5fd48ae06e9c6d,ae5fb51f1de55713] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:09:51.427 INFO 2458 --- [host-check-clockdiff1,7b5fd48ae06e9c6d,26095e2e4961e9da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:09:51.427 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:51.428 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:51.428 INFO 2458 --- [host-check-clockdiff2,7b5fd48ae06e9c6d,f4891f925e10fb69] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:09:51.428 INFO 2458 --- [host-check-clockdiff4,7b5fd48ae06e9c6d,e3a3d4011d8500ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:09:51.428 INFO 2458 --- [host-check-clockdiff3,7b5fd48ae06e9c6d,1442f41505e1c23c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:09:51.429 INFO 2458 --- [host-check-clockdiff5,7b5fd48ae06e9c6d,89c6a3e0a039cebd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:09:51.429 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:51.434 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:51.435 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:51.436 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:51.436 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:51.436 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:51.437 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:51.455 INFO 2458 --- [ocp-schedules-28,2752e0473ad498c2,78808ea73ea42632] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:09:51.455 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:51.455 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:51.456 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:51.457 INFO 2458 --- [host-check-clockdiff1,7b5fd48ae06e9c6d,26095e2e4961e9da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534591 0 0 , error= 2025-10-27 11:09:51.463 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:51.465 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:51.465 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:51.466 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:51.472 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:51.472 INFO 2458 --- [host-check-clockdiff2,7b5fd48ae06e9c6d,f4891f925e10fb69] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534591 -1 -1 , error= 2025-10-27 11:09:51.472 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:51.472 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:51.473 INFO 2458 --- [host-check-clockdiff3,7b5fd48ae06e9c6d,1442f41505e1c23c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534591 -1 -1 , error= 2025-10-27 11:09:51.479 INFO 2458 --- [host-check-clockdiff4,7b5fd48ae06e9c6d,e3a3d4011d8500ba] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534591 0 0 , error= 2025-10-27 11:09:51.528 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:09:51.531 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:09:51.533 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:09:51.532732693+08:00 2025-10-27 11:09:51.534 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:09:51.533+08:00 2025-10-27 11:09:51.535 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:09:51.536 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:09:51.536 INFO 2458 --- [host-check-agent-status2,da103117873344a7,688c879c23de4f1a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:09:51.537 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:09:51.537 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:09:51.539 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:09:51.539 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:09:51.538528913+08:00 2025-10-27 11:09:51.540 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:09:51.540 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:09:51.540 INFO 2458 --- [host-check-agent-status1,da103117873344a7,3a630034b7482200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:09:51.541 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:09:51.539+08:00 2025-10-27 11:09:51.541 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:09:51.540547606+08:00 2025-10-27 11:09:51.541 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:09:51.542 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:09:51.541+08:00 2025-10-27 11:09:51.545 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:09:51.545 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:09:51.545 INFO 2458 --- [host-check-agent-status3,da103117873344a7,5bf2d34bdd094235] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:09:51.545 INFO 2458 --- [ocp-schedules-17,813d5aeac2230d64,b0c54aa5b8127ea0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:09:51.546 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:09:51.547 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:09:51.548 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:09:51.547016530+08:00 2025-10-27 11:09:51.549 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:09:51.549 INFO 2458 --- [ocp-schedules-11,dd2243690cd5f360,aafe2d438836ce27] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:09:51.549 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:09:51.549 INFO 2458 --- [host-check-agent-status5,da103117873344a7,0fc41105ca0221d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:09:51.549 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:09:51.548+08:00 2025-10-27 11:09:51.554 INFO 2458 --- [ocp-schedules-16,4191318af84ce608,1b8d52fb9bc16b7e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:09:51.554 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:09:51.533+08:00 and status AVAILABLE 2025-10-27 11:09:51.554 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:09:51.555 INFO 2458 --- [ocp-schedules-11,dd2243690cd5f360,aafe2d438836ce27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:51.555 INFO 2458 --- [ocp-schedules-17,813d5aeac2230d64,b0c54aa5b8127ea0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:51.556 INFO 2458 --- [ocp-schedules-11,dd2243690cd5f360,aafe2d438836ce27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:09:51.556 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:09:51.556 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:09:51.556 INFO 2458 --- [host-check-agent-status4,da103117873344a7,9c80ac24b5fbbe65] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:09:51.556 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:09:51.559 INFO 2458 --- [ocp-schedules-16,4191318af84ce608,1b8d52fb9bc16b7e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:51.559 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:09:51.559295603+08:00 2025-10-27 11:09:51.559 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:09:51.559 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:09:51.541+08:00 and status AVAILABLE 2025-10-27 11:09:51.560 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:09:51.539+08:00 and status AVAILABLE 2025-10-27 11:09:51.560 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:09:51.559+08:00 2025-10-27 11:09:51.560 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:09:51.548+08:00 and status AVAILABLE 2025-10-27 11:09:51.564 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:09:51.566 INFO 2458 --- [ocp-schedules-24,79227527a5ace610,3c59ce99543b77ca] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:09:51.567 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:09:51.567 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:09:51.568 INFO 2458 --- [ocp-schedules-29,cb39bd0e3c7d0f9e,4f6eccbdc5803809] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:09:51.569 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:09:51.559+08:00 and status AVAILABLE 2025-10-27 11:09:51.569 INFO 2458 --- [ocp-schedules-18,5258c2f683a871c4,2d414c4eeffad5d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:09:51.574 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:09:51.577 INFO 2458 --- [ocp-schedules-29,cb39bd0e3c7d0f9e,4f6eccbdc5803809] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:09:51.578 INFO 2458 --- [ocp-schedules-18,5258c2f683a871c4,2d414c4eeffad5d3] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:09:51.589 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:51.592 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:09:51.593 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:51.595 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:09:51.596 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:09:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:51.598 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:09:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:09:51.601 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:09:51.601 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:51.604 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:09:51.604 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:51.627 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.628 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:09:51.630 INFO 2458 --- [ocp-schedules-17,813d5aeac2230d64,b0c54aa5b8127ea0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:09:51.649 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:51.655 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:51.656 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:51.659 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:51.669 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:09:51.670 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:09:51.676 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:09:51.676 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:09:51.683 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:51.683 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.683 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:51.684 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:09:51.685 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:09:51.692 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:09:51.703 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:09:51.710 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:09:51.714 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:09:51.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:09:51.713635+08:00, lastProcessTime=2025-10-27T11:09:41.717505+08:00 2025-10-27 11:09:51.731 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:09:51.732 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:09:51.741 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:51.746 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:51.759 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:51.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:51.762 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:09:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.763 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:51.764 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:09:51.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:09:51.766 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:51.768 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:51.769 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:09:51.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:51.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.776 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.777 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.777 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.777 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.777 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:09:51.784 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:09:51.785 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:09:51.798 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:09:51.803 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:51.803 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:51.818 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:09:51.818 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:09:51.829 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:09:51.833 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:09:51.835 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:09:51.837 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:51.853 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:51.860 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:51.862 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:51.863 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:51.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:51.866 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.866 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:51.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:09:51.871 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:09:51.874 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:09:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:09:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:51.885 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:09:51.886 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:09:51.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:09:51.887 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:09:51.888 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:51.889 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:09:51.893 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:09:51.907 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:09:51.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:09:51.912 INFO 2458 --- [host-check-host-status5,6f7ec59902de6142,5f3bd587ea61c6d6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:09:51.926 INFO 2458 --- [host-check-host-status2,6f7ec59902de6142,98e770fa995f14ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:09:51.930 INFO 2458 --- [host-check-host-status3,6f7ec59902de6142,ae3e71df09f4664f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:09:51.941 INFO 2458 --- [host-check-host-status4,6f7ec59902de6142,f1e6a1cc9ef1b22d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:09:51.944 INFO 2458 --- [host-check-host-status1,6f7ec59902de6142,e3ea35a55529459c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:09:51.960 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:51.961 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:51.985 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:09:52.001 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:09:51.713635+08:00 2025-10-27 11:09:52.013 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:52.013 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:52.028 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:09:52.028 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.029 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:09:52.031 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.032 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:09:52.048 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:09:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:52.071 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:52.072 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.072 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:09:52.090 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:52.090 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:09:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:09:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:09:52.135 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:52.135 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:09:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:09:52.214 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:52.215 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.215 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.215 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.215 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.215 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:09:52.219 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:09:52.230 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:09:52.240 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:09:52.243 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:09:52.291 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:09:52.343 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:09:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:09:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:09:52.378 INFO 2458 --- [host-check-clockdiff5,7b5fd48ae06e9c6d,89c6a3e0a039cebd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534592 1 1 , error= 2025-10-27 11:09:52.385 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:09:52.424 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:52.424 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.424 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.425 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:52.425 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:52.425 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:52.425 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:09:52.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:09:52.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534002508000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:09:52.463 INFO 2458 --- [ocp-schedules-16,4191318af84ce608,1b8d52fb9bc16b7e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:09:52.513 WARN 2458 --- [ocp-async-4,f5936ba4c0e86712,29fcc0261fb788f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:52.513 INFO 2458 --- [ocp-async-2,0a727e640b53e6f0,f7aa7f0700bdcb00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:09:53.220 INFO 2458 --- [ocp-schedules-9,586a43f6af3f56e5,ff658a05e9725e39] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20309,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:09:53.220 INFO 2458 --- [ocp-schedules-9,586a43f6af3f56e5,ff658a05e9725e39] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:09:53.707 INFO 2458 --- [prometheus-minute-schedule-1,17a22380e4885f43,244c52f87600cf31] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:53.715 INFO 2458 --- [prometheus-minute-schedule-1,17a22380e4885f43,244c52f87600cf31] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:53.757 INFO 2458 --- [ocp-schedules-2,07bde210112f57eb,02d526c05aa3cd74] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:09:53.760 INFO 2458 --- [ocp-schedules-2,07bde210112f57eb,02d526c05aa3cd74] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:09:54.034 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:54.034 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:54.344 INFO 2458 --- [ocp-schedules-10,cba340dd720c1e7a,6c43ee6f32f4401f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:09:54.364 WARN 2458 --- [scanner-112140831-1,cba340dd720c1e7a,52b1095afd0684cd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:09:54.513 WARN 2458 --- [ocp-async-5,0e7b6aeb51bf2b63,b2680276dcc07076] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:54.513 INFO 2458 --- [ocp-async-10,bde85bdcab018276,4efa0868b832b4a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:54.519 INFO 2458 --- [ocp-schedules-22,587e3b814f321a1f,49ed1f7a1f69c78c] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:09:55.939 INFO 2458 --- [ocp-schedules-4,214096485c1bcd23,956940bce0a37f98] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:09:56.513 WARN 2458 --- [ocp-async-3,3b1f979c41758f19,99e9fd1488ab97d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:56.513 INFO 2458 --- [ocp-async-3,a616af37009b9938,1767640fbb2caf32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:56.707 INFO 2458 --- [prometheus-minute-schedule-1,778f61b53797971f,ca7d297a2fcf1c69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:56.715 INFO 2458 --- [prometheus-minute-schedule-1,778f61b53797971f,ca7d297a2fcf1c69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:09:57.035 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:09:57.035 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:09:57.735 INFO 2458 --- [ocp-schedules-5,d5421dae7c140f54,0c9bdaf795e31cb3] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:09:57.742 INFO 2458 --- [ocp-schedules-5,d5421dae7c140f54,0c9bdaf795e31cb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:57.787 INFO 2458 --- [ocp-schedules-5,d5421dae7c140f54,0c9bdaf795e31cb3] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:09:58.511 INFO 2458 --- [ocp-async-1,2e9a62a93fc66d66,37c50bc87755f5f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:09:58.513 INFO 2458 --- [ocp-async-4,d2bd1bf56c045f74,62509dbcaaf176b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:09:58.514 WARN 2458 --- [ocp-async-7,d54481b5b7bfd308,e0429f98776669c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:09:58.828 INFO 2458 --- [ocp-schedules-8,2c6467db69da20b9,a343ae098aad82a9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:09:58.835 INFO 2458 --- [ocp-schedules-8,2c6467db69da20b9,a343ae098aad82a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:09:58.913 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:09:58.914 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.917 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:09:58.926 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.926 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:58.951 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.951 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:58.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:58.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:58.969 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:09:58.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:58.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:58.970 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.973 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:09:58.976 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.976 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:58.978 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.979 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:58.988 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:58.988 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:58.988 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:58.988 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:59.005 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.006 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:59.017 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.017 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.017 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.018 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:59.032 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.032 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:59.046 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.046 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.046 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.047 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:09:59.049 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.049 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.049 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.050 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:59.062 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.063 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.064 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.065 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.065 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.066 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.067 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.068 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.069 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.070 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.070 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.072 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.072 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:09:59.075 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:09:59.079 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.079 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.079 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.080 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:59.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.108 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:09:59.111 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.111 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.111 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.112 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:09:59.127 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:59.128 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.128 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:09:59.138 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.138 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:09:59.138 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.138 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:09:59.141 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:09:59.141 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:09:59.147 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.147 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.150 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.151 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:09:59.184 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.184 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.184 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.184 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:09:59.187 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.187 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.187 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.187 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:09:59.212 INFO 2458 --- [ocp-schedules-20,b9a8c7f11847d317,5ee11a41c93d4138] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:09:59.215 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.215 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.215 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.216 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:09:59.244 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.244 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.245 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.245 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:09:59.291 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.291 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.291 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.292 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.293 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.293 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.293 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.294 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.294 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.294 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.294 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.295 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.295 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.295 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.295 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.296 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.296 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.296 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.297 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.297 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.298 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.298 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.298 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.299 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.299 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.299 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.299 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.300 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.301 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.301 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.301 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.302 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.302 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.302 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.302 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.303 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.304 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.304 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.305 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:09:59.305 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.305 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.305 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.306 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:09:59.306 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.306 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.307 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.307 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:09:59.308 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:09:59.311 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:09:59.397 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:09:59.398 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.398 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.398 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:09:59.398 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:09:59.398 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:09:59.399 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:09:59.424 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:09:59.428 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:09:59.428 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:09:59.428 INFO 2458 --- [ocp-schedules-8,2c6467db69da20b9,a343ae098aad82a9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:09:59.579 INFO 2458 --- [ocp-schedules-6,8b4da2cd1fa0ab1e,ca29ed27f62fe708] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:09:59.583 INFO 2458 --- [ocp-schedules-6,8b4da2cd1fa0ab1e,ca29ed27f62fe708] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:09:59.707 INFO 2458 --- [prometheus-minute-schedule-1,bde9981334c2c1ce,859068f56346592e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534600, interval=60 2025-10-27 11:09:59.714 INFO 2458 --- [prometheus-minute-schedule-1,bde9981334c2c1ce,859068f56346592e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:00.035 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:00.036 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:00.514 WARN 2458 --- [ocp-async-5,bc649bb5e0feb0d3,4c2767584154ab98] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:00.514 INFO 2458 --- [ocp-async-8,46fd5d445a7a1d63,98b052b5345812f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:00.746 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:05:00.0, nextRun=2025-10-27 11:10:00.0, currentDate=Mon Oct 27 11:10:00 CST 2025 2025-10-27 11:10:00.746 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:05:00.0, nextRunTime=2025-10-27 11:10:00.0} 2025-10-27 11:10:00.759 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:10:00.759 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:10:00.759+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:10:00.785 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:05:00.0, nextRun=2025-10-27 11:10:00.0, currentDate=Mon Oct 27 11:10:00 CST 2025 2025-10-27 11:10:00.785 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:05:00.0, nextRunTime=2025-10-27 11:10:00.0} 2025-10-27 11:10:00.840 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:10:00.879 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:10:00.885 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:10:00.885 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,3e273e5aa1ebb520] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:10:00.885+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:10:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:01.715638+08:00, lastProcessTime=2025-10-27T11:09:51.713635+08:00 2025-10-27 11:10:01.780 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,6ff6e36ceac79765] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9155, operation:EXECUTE, state: RUNNING 2025-10-27 11:10:01.791 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,6ff6e36ceac79765] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9156, operation:EXECUTE, state: RUNNING 2025-10-27 11:10:01.815 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:10:01.816 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9155, context=Context{parallelIdx=-1, stringMap={task_instance_id=9097, task_operation=execute, latest_execution_start_time=2025-10-27T11:10:01.767+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9155}, listMap={}}, executor=10.40.65.146 2025-10-27 11:10:01.832 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:10:01.836 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9156, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9098, task_operation=execute, latest_execution_start_time=2025-10-27T11:10:01.789+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9156}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:10:01.837 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:10:01.902 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:01.977 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:01.715638+08:00 2025-10-27 11:10:01.997 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.000 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:10:02.063 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.123 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:10:02.130 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.253 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:10:02.254 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.255 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:10:02.323 INFO 2458 --- [ocp-schedules-26,4dc6643112dac032,7a19d76e01b11613] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:10:02.339 INFO 2458 --- [ocp-schedules-26,4dc6643112dac032,7a19d76e01b11613] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:02.372 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.387 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:10:02.389 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:02.390 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:10:02.390 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:10:02.390 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:10:02.418 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:02.418 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:10:02.437 INFO 2458 --- [ocp-schedules-26,4dc6643112dac032,7a19d76e01b11613] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:10:02.494 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:10:02.502 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:10:02.503 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,1e9e46b5f1187e40,b635559c9c5ee207] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&version=2&ObClusterId=1760510187, client=10.40.65.146, traceId=1e9e46b5f1187e40, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase,1760510187,2,{"ObRegion":"myoceanbase","ObCluster":"myoceanbase","ObRegionId":1760510187,"ObClusterId":1760510187,"Type":"PRIMARY","timestamp":1761534602500979,"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:10:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,1e9e46b5f1187e40,b635559c9c5ee207] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase, obClusterId=1760510187, rootServerJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, groupName=myoceanbase, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:10:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,1e9e46b5f1187e40,b635559c9c5ee207] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:10:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,1e9e46b5f1187e40,b635559c9c5ee207] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase, clusterId=1760510187, affectRows=1 2025-10-27 11:10:02.514 INFO 2458 --- [ocp-async-6,ef98f7c08e144cd4,1c6ef230e32b38cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:02.514 WARN 2458 --- [ocp-async-9,4013b854329ecba6,0da88f0e32dec201] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,1e9e46b5f1187e40,b635559c9c5ee207] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.65.146, traceId=1e9e46b5f1187e40, duration=12 ms] 2025-10-27 11:10:02.526 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,b6df5a751e131fce] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9156, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:10:02.667 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.707 INFO 2458 --- [prometheus-minute-schedule-1,cb56c8c11f3a3e07,ec1016a2a8ac0454] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:02.710 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9098, status:SUCCESS 2025-10-27 11:10:02.723 INFO 2458 --- [prometheus-minute-schedule-1,cb56c8c11f3a3e07,ec1016a2a8ac0454] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:02.756 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9098, status:SUCCESS 2025-10-27 11:10:02.769 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:10:02.793 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:10:02.793 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:10:02.800 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:02.820 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:02.838 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:02.860 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:02.921 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:02.970 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:03.036 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:03.037 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:03.061 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:03.088 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:10:03.106 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:10:03.106 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:10:03.128 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,7db32d681cb6a966] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9155, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:10:03.768 INFO 2458 --- [ocp-schedules-30,67702a043336817e,50613cf7cdc8ff66] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:03.771 INFO 2458 --- [ocp-schedules-30,67702a043336817e,50613cf7cdc8ff66] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:03.795 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9097, status:SUCCESS 2025-10-27 11:10:03.811 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9097, status:SUCCESS 2025-10-27 11:10:04.022 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,102e80fdc5ca45e2] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:00:04.0, nextRun=2025-10-27 11:10:04.0, currentDate=Mon Oct 27 11:10:04 CST 2025 2025-10-27 11:10:04.022 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,102e80fdc5ca45e2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-10-27 11:00:04.0, nextRunTime=2025-10-27 11:10:04.0} 2025-10-27 11:10:04.035 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,102e80fdc5ca45e2] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-10-27 11:10:04.035 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,102e80fdc5ca45e2] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:10:04.035+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:10:04.455 INFO 2458 --- [ocp-schedules-26,98d5d5f6715889e3,4e1bcf5718f12373] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:04.510 WARN 2458 --- [scanner-112141172-1,98d5d5f6715889e3,273bd54408d4b75b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:04.514 INFO 2458 --- [ocp-async-2,e67337a8fa80ba36,10ac5a890684298c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:04.514 WARN 2458 --- [ocp-async-1,c9dad23d0c0e7b7d,fd9f7d7dd6eb4a8c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:04.687 INFO 2458 --- [alarm-detect-11,98d5d5f6715889e3,6b24ebf8159ec00f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534570 ~ 1761534594 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534585, value=43.462)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534585, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534585=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:10:04.856 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,3548380d78a74c54] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9157, operation:EXECUTE, state: RUNNING 2025-10-27 11:10:04.889 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:10:04.890 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9157, context=Context{parallelIdx=-1, stringMap={task_instance_id=9099, task_operation=execute, latest_execution_start_time=2025-10-27T11:10:04.852+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=9157}, listMap={}}, executor=10.40.65.146 2025-10-27 11:10:04.899 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:10:04.899 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:10:04.902 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.903 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:10:04.906 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.908 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.909 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.910 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:10:04.927 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.927 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.928 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.929 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:10:04.934 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:10:04.934 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:10:04.935 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.935 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:10:04.937 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.938 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.939 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:10:04.958 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.958 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.959 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.960 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:10:04.977 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.977 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.978 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.979 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:10:04.994 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.994 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:04.995 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:04.996 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:10:05.009 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.010 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.011 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.011 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:10:05.029 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.030 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.031 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.031 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:10:05.046 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.047 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.048 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.048 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:10:05.064 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.064 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.065 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.066 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:10:05.073 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.073 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.074 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.075 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:10:05.083 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.084 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.085 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.085 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:10:05.091 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.091 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.092 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.093 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:10:05.099 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.100 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.101 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.101 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:10:05.108 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.109 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.110 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.110 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:10:05.118 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.119 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.120 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.120 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:10:05.127 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.127 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.128 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.129 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:10:05.135 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.136 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.137 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.137 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:10:05.144 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.144 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.145 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.146 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:10:05.152 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.153 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.154 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.154 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:10:05.164 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.165 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.165 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.166 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:10:05.175 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.176 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.177 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.177 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:10:05.191 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.192 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.193 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.193 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:10:05.199 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.200 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.201 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.201 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:10:05.208 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.209 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.209 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.210 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:10:05.216 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.216 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.217 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.218 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:10:05.224 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.224 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.225 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.225 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:10:05.232 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.232 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.233 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.233 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:10:05.239 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.240 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.241 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.241 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:10:05.247 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.248 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.248 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.249 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:10:05.254 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.255 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.256 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.256 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:10:05.262 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.262 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.263 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.264 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:10:05.269 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.269 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.270 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.271 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:10:05.276 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.277 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.278 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.278 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:10:05.284 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.284 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.285 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.286 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:10:05.291 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.291 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.292 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.293 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:10:05.298 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.299 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.300 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.300 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:10:05.306 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.306 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.307 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.307 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:10:05.313 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.313 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.314 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.315 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:10:05.320 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.321 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.322 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.322 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:10:05.327 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.328 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.329 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.329 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:10:05.335 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.335 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.336 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.337 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:10:05.342 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.343 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.343 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.344 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:10:05.349 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.350 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.350 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.351 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:10:05.362 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.363 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.364 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.364 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:10:05.371 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.372 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.373 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.373 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:10:05.378 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.379 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.380 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.380 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:10:05.385 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.386 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.387 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.388 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:10:05.393 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.394 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.395 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.395 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:10:05.400 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.401 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.402 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.403 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:10:05.407 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.408 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:10:05.414 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.415 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:10:05.423 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.424 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.424 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.425 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:10:05.430 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.431 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.432 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.432 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:10:05.437 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.438 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.439 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.439 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:10:05.444 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.445 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:10:05.451 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.452 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.453 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.453 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:10:05.457 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.458 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.458 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.459 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:10:05.464 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.465 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:05.466 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:05.467 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:10:05.471 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:10:05.493 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ad700ae8c42f30fc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9157, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:10:05.707 INFO 2458 --- [prometheus-minute-schedule-1,8ef67b8b883a6414,f9c8d61e172d00b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:05.718 INFO 2458 --- [prometheus-minute-schedule-1,8ef67b8b883a6414,f9c8d61e172d00b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:05.853 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9099, status:SUCCESS 2025-10-27 11:10:05.856 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9099, status:SUCCESS 2025-10-27 11:10:05.945 INFO 2458 --- [ocp-schedules-12,39fd67adddbd04f9,7a4808f74e38f12f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:10:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:10:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:10:06.037 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:06.038 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:06.064 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:10:06.105 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:10:06.131 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:10:06.131 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:10:06.131 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:10:06.131 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:10:06.135 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:10:06.136 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:10:06.137 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:06.138 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:10:06.208 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:07:30, start=2025-10-27T11:05:00, end=2025-10-27T11:07:30, updates=263, granule=120000ms, delay=150000ms 2025-10-27 11:10:06.216 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:07:30, target=2025-10-27T11:07:30, granule=120000ms, delay=0ms) 2025-10-27 11:10:06.342 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - source=2025-10-27T11:07:30, start=2025-10-27T10:52:30, end=2025-10-27T11:02:30, updates=390, granule=600000ms, delay=900000ms 2025-10-27 11:10:06.353 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=1) - not enough cached interval (source=2025-10-27T11:07:30, target=2025-10-27T11:02:30, granule=600000ms, delay=300000ms) 2025-10-27 11:10:06.353 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=349ms, period=120000ms 2025-10-27 11:10:06.514 WARN 2458 --- [ocp-async-8,f8f55fc4a6ebc268,93c19c935f2de6f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:06.514 INFO 2458 --- [ocp-async-5,2a3bf6baf8c37906,18e393b1583a45b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:06.807 INFO 2458 --- [ocp-schedules-13,9969285008850ac8,b3e3162f4042ecfc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:10:06.816 INFO 2458 --- [ocp-schedules-13,9969285008850ac8,b3e3162f4042ecfc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:06.816 INFO 2458 --- [ocp-schedules-13,9969285008850ac8,b3e3162f4042ecfc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:08.512 INFO 2458 --- [ocp-async-3,ecd57d0262ca0c02,e4f66de014b92456] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:08.514 INFO 2458 --- [ocp-async-9,4e1777c34209bdde,dba0098aabd7b948] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:08.514 WARN 2458 --- [ocp-async-7,d6a4e6606359d422,fded4523b0acde22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:08.707 INFO 2458 --- [prometheus-minute-schedule-1,4e2a5e7cb5471b16,7e37bef31ca1eb4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:08.717 INFO 2458 --- [prometheus-minute-schedule-1,4e2a5e7cb5471b16,7e37bef31ca1eb4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:09.038 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:09.039 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:10.024 INFO 2458 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:10:10.024 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:10:10.080 INFO 2458 --- [Collect obproxy1,7d5e4e17c3a43110,16a5117434ff87be] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:10:10.080 INFO 2458 --- [Collect obproxy1,7d5e4e17c3a43110,16a5117434ff87be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:10:10.082 INFO 2458 --- [Check obproxy1,39665a9e423ed563,97863e3f9ca579de] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:10:10.082 INFO 2458 --- [Check obproxy1,39665a9e423ed563,97863e3f9ca579de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:10:10.111 INFO 2458 --- [Collect obproxy1,7d5e4e17c3a43110,16a5117434ff87be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:10:10.114 INFO 2458 --- [Collect obproxy1,7d5e4e17c3a43110,16a5117434ff87be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:10:10.116 INFO 2458 --- [Check obproxy1,39665a9e423ed563,97863e3f9ca579de] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:10:10.117 INFO 2458 --- [Collect obproxy1,7d5e4e17c3a43110,16a5117434ff87be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:10:10.119 INFO 2458 --- [Check obproxy1,39665a9e423ed563,97863e3f9ca579de] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:10:10.119 INFO 2458 --- [Check obproxy1,39665a9e423ed563,97863e3f9ca579de] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:10:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:10:10.514 INFO 2458 --- [ocp-async-4,af43a8e89da7ff84,cfe3aa1daf6bef68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:10.515 WARN 2458 --- [ocp-async-2,3855fe341e6b0090,856f82b2578ec480] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:11.708 INFO 2458 --- [prometheus-minute-schedule-1,6bcbfa3195b7eab1,e58dbd5d25523217] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:11.715 INFO 2458 --- [prometheus-minute-schedule-1,6bcbfa3195b7eab1,e58dbd5d25523217] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:11.747 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:11.746075+08:00, lastProcessTime=2025-10-27T11:10:01.715638+08:00 2025-10-27 11:10:11.981 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:11.746075+08:00 2025-10-27 11:10:12.039 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:12.040 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:12.515 WARN 2458 --- [ocp-async-8,10fa30addd1ba648,d9bd3594f634c1f9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:12.515 INFO 2458 --- [ocp-async-8,c605c0c0edcd62ac,f5c48a55e12587a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:13.660 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534540, nodeCount=405, affectRows=405 2025-10-27 11:10:13.666 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534540, nodeCount=412, affectRows=412 2025-10-27 11:10:13.779 INFO 2458 --- [ocp-schedules-15,8119c128d36113ea,e4bf9b343e708bd8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:13.783 INFO 2458 --- [ocp-schedules-15,8119c128d36113ea,e4bf9b343e708bd8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:13.806 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=1000, affectRows=1000 2025-10-27 11:10:13.810 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=994, validCount=992 2025-10-27 11:10:13.825 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134601996) Duplicate entry '835370602979598336-1761534480' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134601996) Duplicate entry '835370602979598336-1761534480' for key 'PRIMARY' 2025-10-27 11:10:13.912 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=1000, affectRows=1000 2025-10-27 11:10:13.916 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=999, validCount=995 2025-10-27 11:10:14.026 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=1000, affectRows=1000 2025-10-27 11:10:14.030 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=992, validCount=988 2025-10-27 11:10:14.102 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=1000, affectRows=1000 2025-10-27 11:10:14.106 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=1000, validCount=996 2025-10-27 11:10:14.121 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=77, affectRows=77 2025-10-27 11:10:14.121 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=77, validCount=77 2025-10-27 11:10:14.385 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38} 2025-10-27 11:10:14.397 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClust 2025-10-27 11:10:14.397 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:14.408 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 33 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:14.479 INFO 2458 --- [ocp-schedules-9,5038154b1b0de618,46e8383f615c7d97] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:10:14.479+08:00, clusters=[1, 2] 2025-10-27 11:10:14.515 INFO 2458 --- [ocp-async-6,fc2bddaf0bbb8bad,60b8faca647fcff9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:14.515 WARN 2458 --- [ocp-async-3,74355140aee05c93,df12cda0f4851595] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:14.532 INFO 2458 --- [ocp-schedules-9,5038154b1b0de618,46e8383f615c7d97] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:10:14.542 INFO 2458 --- [ocp-schedules-9,5038154b1b0de618,46e8383f615c7d97] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:10:14.561 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:10:14.561 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:14.597 INFO 2458 --- [ocp-schedules-21,ad4df8fd9d389788,703be623fce61fb8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:10:14.597 INFO 2458 --- [ocp-schedules-21,ad4df8fd9d389788,703be623fce61fb8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:10:14.656 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:10:14.656 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:10:14.657 INFO 2458 --- [Collect all tenants major compactions5,ad4df8fd9d389788,159c4434b19d334f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:10:14.657 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:10:14.658 INFO 2458 --- [Collect all tenants major compactions6,ad4df8fd9d389788,c931d0f4a73dc7e4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:10:14.658 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:10:14.694 INFO 2458 --- [ocp-schedules-12,fbba2c8d0cb4b32d,cbe525eb40a959ab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:14.701 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448214701000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:10:14.701Z, endTime=2025-10-27T03:08:14.380Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:10:14.705 INFO 2458 --- [Collect all tenants major compactions5,ad4df8fd9d389788,159c4434b19d334f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.707 INFO 2458 --- [Collect all tenants major compactions5,ad4df8fd9d389788,159c4434b19d334f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:10:14.707 INFO 2458 --- [Collect all tenants major compactions6,ad4df8fd9d389788,c931d0f4a73dc7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.708 INFO 2458 --- [prometheus-minute-schedule-1,3cc724b9ba05a3c6,6ba9441e774fa8ee] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:14.710 INFO 2458 --- [Collect all tenants major compactions6,ad4df8fd9d389788,c931d0f4a73dc7e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:10:14.713 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.713 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions5,ad4df8fd9d389788,159c4434b19d334f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.714 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.715 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.715 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.715 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.717 INFO 2458 --- [prometheus-minute-schedule-1,3cc724b9ba05a3c6,6ba9441e774fa8ee] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:14.717 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:10:14.718 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:10:14.722 INFO 2458 --- [Collect all tenants major compactions2,ad4df8fd9d389788,5416e1a31e793941] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:10:14.722 INFO 2458 --- [Collect all tenants major compactions1,ad4df8fd9d389788,ad88c2ed60c96507] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:10:14.726 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:10:14.726 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:10:14.743 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.743 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.744 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.754 WARN 2458 --- [scanner-112142988-1,fbba2c8d0cb4b32d,549907cb7b379691] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:14.715 INFO 2458 --- [Collect all tenants major compactions6,ad4df8fd9d389788,c931d0f4a73dc7e4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:10:14.773 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:10:14.773 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:10:14.744 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.776 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.776 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.776 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.776 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:14.776 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:14.777 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:14.778 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:10:14.781 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:10:14.781 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:10:14.781 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:10:14.782 INFO 2458 --- [Collect all tenants major compactions4,ad4df8fd9d389788,a1b59688280d7a61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:10:14.785 INFO 2458 --- [Collect all tenants major compactions3,ad4df8fd9d389788,6328a5ee39495a8d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:10:14.807 INFO 2458 --- [ocp-schedules-21,ad4df8fd9d389788,703be623fce61fb8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:10:14.817 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:10:14.817 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:10:14.822 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:10:14.822 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:10:14.701, end=2025-10-27T11:08:14.38}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:10:14.823 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:14.824 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:14.825 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:10:14.825 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:14.828 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:14.829 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:14.829 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:10:14.829 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:14.832 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:14.836 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:14.836 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:10:14.836 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:14.871 INFO 2458 --- [alarm-detect-15,fbba2c8d0cb4b32d,d2193c20caaaba48] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534585 ~ 1761534604 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534600, value=54.88)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534600, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534600=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:10:15.000 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448215000000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:10:15Z, endTime=2025-10-27T03:08:14.380Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:10:15.040 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:15.041 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:15.122 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 260 2025-10-27 11:10:15.122 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:10:15.167 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:10:15.167 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:10:15.176 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:10:15.176 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:10:15.201 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:10:15.201 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534494380000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:10:15.206 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:10:15.233 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:10:15, end=2025-10-27T11:08:14.38}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:10:15.234 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:15.235 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:15.235 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:10:15.235 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:15.238 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:10:15.239 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:13.702, end=2025-10-27T11:08:14.38},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:10:15.239 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:10:15.239 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:10:15.241 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:10:15.877 INFO 2458 --- [ocp-schedules-3,839bb652352efa9e,eb7bddf9763602cd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:10:15.877 INFO 2458 --- [ocp-schedules-3,839bb652352efa9e,eb7bddf9763602cd] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:10:15.888 INFO 2458 --- [ocp-schedules-3,839bb652352efa9e,eb7bddf9763602cd] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:10:15.888 INFO 2458 --- [ocp-schedules-3,839bb652352efa9e,eb7bddf9763602cd] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:10:15.953 INFO 2458 --- [ocp-schedules-10,8f8f7c2a269ae456,894f937d7d3a8564] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:16.515 INFO 2458 --- [ocp-async-9,34270c24109c21bc,81d3c1f8ca1b7453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:16.515 WARN 2458 --- [ocp-async-2,be873714ee38d75c,b3f2c75cbbd859af] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:17.707 INFO 2458 --- [prometheus-minute-schedule-1,90f684f03988029d,056375b0806b42bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:17.718 INFO 2458 --- [prometheus-minute-schedule-1,90f684f03988029d,056375b0806b42bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:18.041 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:18.042 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:18.314 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534540, nodeCount=1000, affectRows=1000 2025-10-27 11:10:18.318 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534540, nodeCount=999, validCount=986 2025-10-27 11:10:18.512 INFO 2458 --- [ocp-async-9,b10b27eebaef3343,bac46e5cf96af624] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:18.515 INFO 2458 --- [ocp-async-4,5e4bd3061b2fa727,fddfc15304ec40f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:18.515 WARN 2458 --- [ocp-async-10,247a95ea68c76df3,9e57f8123b85312a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:18.833 INFO 2458 --- [ocp-schedules-19,4e8a9af04e9f1af4,7369a5aaaa2e0bc9] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:18.833 INFO 2458 --- [ocp-schedules-19,4e8a9af04e9f1af4,7369a5aaaa2e0bc9] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:10:19.344 INFO 2458 --- [ocp-schedules-9,f0b42d228ec1ba91,77a84ebab4938e50] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:10:20.515 INFO 2458 --- [ocp-async-4,823840ed91070ffa,db241564fd2afd48] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:20.515 WARN 2458 --- [ocp-async-3,c2620ac325593a4e,d636ee0b91ea5fae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:20.707 INFO 2458 --- [prometheus-minute-schedule-1,667f547164404081,005114906a57c1aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:20.716 INFO 2458 --- [prometheus-minute-schedule-1,667f547164404081,005114906a57c1aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:21.043 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:21.043 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:21.411 INFO 2458 --- [ocp-schedules-21,0a108b450ec313ee,89632c3316b56989] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:21.411 INFO 2458 --- [ocp-schedules-3,63754fe27ed3be4d,411f7621d234ba80] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:21.412 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:10:21.412 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:10:21.412 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:10:21.412 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:10:21.412 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:10:21.413 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:10:21.413 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:10:21.413 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:10:21.413 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:10:21.414 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:10:21.426 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:21.426 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:21.427 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:21.427 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:21.428 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:21.428 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:21.428 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:21.429 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:21.432 INFO 2458 --- [ocp-schedules-10,998677ca6e4b9310,bc444928081c1504] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:21.432 INFO 2458 --- [host-check-clockdiff1,998677ca6e4b9310,4bd9b3648d581881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:10:21.432 INFO 2458 --- [host-check-clockdiff2,998677ca6e4b9310,c49a7998f868fa20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:10:21.435 INFO 2458 --- [host-check-clockdiff3,998677ca6e4b9310,406c7449c3b21025] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:10:21.435 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:21.435 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:21.436 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:21.440 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:21.441 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:21.441 INFO 2458 --- [host-check-clockdiff4,998677ca6e4b9310,3038b31baf4c04c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:10:21.442 INFO 2458 --- [ocp-schedules-7,3630249a6e6d59d6,a7b9ba0e9a3f62b2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:10:21.442 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:21.442 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:21.458 INFO 2458 --- [host-check-clockdiff1,998677ca6e4b9310,4bd9b3648d581881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534621 0 0 , error= 2025-10-27 11:10:21.462 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:21.462 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:21.458 INFO 2458 --- [host-check-clockdiff5,998677ca6e4b9310,7deabd653e742e33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:10:21.472 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:21.473 INFO 2458 --- [host-check-clockdiff3,998677ca6e4b9310,406c7449c3b21025] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534621 -1 -1 , error= 2025-10-27 11:10:21.473 INFO 2458 --- [host-check-clockdiff4,998677ca6e4b9310,3038b31baf4c04c3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534621 0 0 , error= 2025-10-27 11:10:21.477 INFO 2458 --- [host-check-clockdiff5,998677ca6e4b9310,7deabd653e742e33] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534621 0 0 , error= 2025-10-27 11:10:21.479 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:21.479 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:21.545 INFO 2458 --- [ocp-schedules-4,4c370997226210f0,98a2cbcde005e366] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:10:21.549 INFO 2458 --- [ocp-schedules-23,708b734f9f93ff0b,2603535b5e039e27] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:10:21.553 INFO 2458 --- [ocp-schedules-4,4c370997226210f0,98a2cbcde005e366] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:21.554 INFO 2458 --- [ocp-schedules-20,10555570a51e56a7,2d69a0dcd4216448] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:10:21.554 INFO 2458 --- [ocp-schedules-23,708b734f9f93ff0b,2603535b5e039e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:21.559 INFO 2458 --- [ocp-schedules-23,708b734f9f93ff0b,2603535b5e039e27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:10:21.560 INFO 2458 --- [ocp-schedules-20,10555570a51e56a7,2d69a0dcd4216448] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:21.576 INFO 2458 --- [ocp-schedules-25,079945b4d523584c,95f7de58963bd581] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:10:21.622 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.623 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:21.624 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:10:21.627 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:10:21.628 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:10:21.629 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:10:21.629084974+08:00 2025-10-27 11:10:21.631 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:10:21.629+08:00 2025-10-27 11:10:21.631 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:10:21.631 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:10:21.631 INFO 2458 --- [host-check-agent-status3,63754fe27ed3be4d,b2c7a27aaaf9379a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:10:21.633 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:10:21.635 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:10:21.635 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.636 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:21.637 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:10:21.637 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:10:21.637 INFO 2458 --- [host-check-agent-status2,63754fe27ed3be4d,8776f44ee999927d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:10:21.638 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:10:21.638 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:10:21.638 INFO 2458 --- [host-check-agent-status1,63754fe27ed3be4d,de301a283dc85d77] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:10:21.641 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:10:21.629+08:00 and status AVAILABLE 2025-10-27 11:10:21.643 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:10:21.644 INFO 2458 --- [ocp-schedules-4,4c370997226210f0,98a2cbcde005e366] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:10:21.645 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:10:21.648 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:10:21.647807693+08:00 2025-10-27 11:10:21.650 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:10:21.648+08:00 2025-10-27 11:10:21.652 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:10:21.653 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:10:21.653 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:10:21.653920802+08:00 2025-10-27 11:10:21.654 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:10:21.654+08:00 2025-10-27 11:10:21.656 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:10:21.658 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:10:21.658 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:10:21.658 INFO 2458 --- [host-check-agent-status4,63754fe27ed3be4d,16d5b4fc2fb2d2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:10:21.660 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:10:21.661 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:10:21.660079990+08:00 2025-10-27 11:10:21.662 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:10:21.661+08:00 2025-10-27 11:10:21.669 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:10:21.648+08:00 and status AVAILABLE 2025-10-27 11:10:21.669 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:10:21.661+08:00 and status AVAILABLE 2025-10-27 11:10:21.669 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:10:21.654+08:00 and status AVAILABLE 2025-10-27 11:10:21.670 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:10:21.672 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:10:21.671670444+08:00 2025-10-27 11:10:21.673 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:10:21.673 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:10:21.672+08:00 2025-10-27 11:10:21.675 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:10:21.675 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:10:21.675 INFO 2458 --- [host-check-agent-status5,63754fe27ed3be4d,74c2ec4604133a09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:10:21.676 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:10:21.679 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:10:21.679 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:10:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:21.686 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:10:21.672+08:00 and status AVAILABLE 2025-10-27 11:10:21.694 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:10:21.699 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:21.700 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:21.702 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:21.703 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:21.714 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.715 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:21.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:21.715988+08:00, lastProcessTime=2025-10-27T11:10:11.746075+08:00 2025-10-27 11:10:21.719 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:21.720 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:21.723 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:21.723 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:21.723 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:21.726 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:21.769 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:21.775 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:21.782 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:21.783 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.783 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:10:21.786 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:21.789 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:21.790 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:10:21.790 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:10:21.797 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:21.809 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:10:21.809 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:21.810 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.810 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:10:21.814 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:10:21.818 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:10:21.838 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:10:21.841 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:10:21.851 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:10:21.862 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:10:21.864 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:10:21.872 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:21.873 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:21.875 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:10:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:21.909 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:21.909 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:21.913 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:21.914 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:21.917 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:21.925 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:10:21.931 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:10:21.933 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:10:21.934 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:10:21.940 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:21.941 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.941 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:10:21.947 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:10:21.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:21.955 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:21.958 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:10:21.966 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:10:21.969 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:10:21.969 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:10:21.979 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:21.980 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:21.981 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:21.982 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:10:21.984 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:10:21.989 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:21.989 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:22.001 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:10:22.006 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:22.010 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:22.013 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:22.014 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:22.015 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:22.020 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:22.022 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:10:22.026 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:22.027 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:10:22.036 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:10:22.036 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:10:22.040 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:22.040 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:22.058 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:10:22.067 INFO 2458 --- [host-check-host-status5,0a108b450ec313ee,4177cd4f7ac73d9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:10:22.067 INFO 2458 --- [host-check-host-status4,0a108b450ec313ee,68b3b799babc143a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:10:22.067 INFO 2458 --- [host-check-host-status3,0a108b450ec313ee,d752c2cf0340c6bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:10:22.074 INFO 2458 --- [host-check-host-status2,0a108b450ec313ee,4d1b1aa46f225464] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:10:22.081 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:22.081 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:22.088 INFO 2458 --- [host-check-host-status1,0a108b450ec313ee,a44fc0a5fd17b0be] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:10:22.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:10:22.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:22.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:22.139 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:10:22.139 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.140 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:10:22.142 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.143 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:10:22.161 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:10:22.164 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:22.164 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:22.184 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:22.184 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.185 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:10:22.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:22.211 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:10:22.211 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:10:22.242 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:22.250 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:10:22.260 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:10:22.269 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:22.272 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:22.330 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:10:22.371 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:21.715988+08:00 2025-10-27 11:10:22.382 INFO 2458 --- [host-check-clockdiff2,998677ca6e4b9310,c49a7998f868fa20] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534622 -1 -1 , error= 2025-10-27 11:10:22.402 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:10:22.403 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.403 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:10:22.405 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.406 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:10:22.442 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:10:22.497 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:22.498 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:22.498 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:10:22.515 INFO 2458 --- [ocp-async-4,30bd66a797241b63,cfb160999421e415] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:22.515 WARN 2458 --- [ocp-async-6,9101089cf331ff01,e3440352f072ec3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:22.537 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:10:22.537 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:10:22.538 INFO 2458 --- [ocp-schedules-20,10555570a51e56a7,2d69a0dcd4216448] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:10:23.707 INFO 2458 --- [prometheus-minute-schedule-1,9dea96ed68d65dbc,211b6dc6cd296bdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:23.715 INFO 2458 --- [prometheus-minute-schedule-1,9dea96ed68d65dbc,211b6dc6cd296bdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:23.792 INFO 2458 --- [ocp-schedules-26,4d445838b84597ae,29de9c958841387d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:23.795 INFO 2458 --- [ocp-schedules-26,4d445838b84597ae,29de9c958841387d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:24.044 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:24.045 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:24.515 INFO 2458 --- [ocp-async-4,00cd68bf996ece0a,6575411d5295328a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:24.516 WARN 2458 --- [ocp-async-5,890d27683c4d4067,47727d048001bc8b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:24.879 INFO 2458 --- [ocp-schedules-22,a0bc211783841ec6,cd044523a22c16ef] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:24.907 WARN 2458 --- [scanner-112143152-1,a0bc211783841ec6,7854889b19fd8866] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:25.063 INFO 2458 --- [alarm-detect-13,a0bc211783841ec6,31be64080de782ca] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534600 ~ 1761534615 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534615, value=86.26)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534615, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534615=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:10:25.965 INFO 2458 --- [ocp-schedules-23,f37191850b41a640,15f97679441f8c5f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:26.516 WARN 2458 --- [ocp-async-3,06aee17d367869a4,61331ad880b8a362] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:26.516 INFO 2458 --- [ocp-async-2,55012c745e7dd1aa,0c5f0ab7bfa34ce7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:26.707 INFO 2458 --- [prometheus-minute-schedule-1,ae43e4a4531036dd,80f4d2fc72d4fc07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:26.714 INFO 2458 --- [prometheus-minute-schedule-1,ae43e4a4531036dd,80f4d2fc72d4fc07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:26.824 INFO 2458 --- [ocp-schedules-19,2918907f8f2f2340,ac3bac250292dd1f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:10:26.833 INFO 2458 --- [ocp-schedules-19,2918907f8f2f2340,ac3bac250292dd1f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:26.833 INFO 2458 --- [ocp-schedules-19,2918907f8f2f2340,ac3bac250292dd1f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:27.045 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:27.047 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:28.512 INFO 2458 --- [ocp-async-2,8b53db2aff11cd37,0eb842d65d97d659] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:28.516 WARN 2458 --- [ocp-async-6,200b53a3202fb8c2,e014b6affebca4a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:28.516 INFO 2458 --- [ocp-async-6,9b6c4a10147be662,b3a7f1b8220598bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:28.867 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:10:28.867 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:10:28.869 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:10:28.869 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:10:28.870 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:10:28.870 INFO 2458 --- [ocp-schedules-8,d4b7edbff8654e18,5374ef859eb0d804] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:10:29.329 INFO 2458 --- [ocp-schedules-22,2fa9e95236d994e6,0f7d25618a12d3dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:10:29.342 INFO 2458 --- [ocp-schedules-22,2fa9e95236d994e6,0f7d25618a12d3dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:10:29.342 INFO 2458 --- [ocp-schedules-22,2fa9e95236d994e6,0f7d25618a12d3dc] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:10:29.707 INFO 2458 --- [prometheus-minute-schedule-1,aa55dde206a4e6e3,5fc596d41369f127] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:29.714 INFO 2458 --- [prometheus-minute-schedule-1,aa55dde206a4e6e3,5fc596d41369f127] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:30.047 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:30.048 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:30.451 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,24a677729d2af847,6a6b5823382e61ba] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=24a677729d2af847, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:10:30.451 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,24a677729d2af847,6a6b5823382e61ba] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=24a677729d2af847, duration=1 ms] 2025-10-27 11:10:30.516 WARN 2458 --- [ocp-async-6,c1416883fb1564db,f2829ed6f3cd3bd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:30.516 INFO 2458 --- [ocp-async-8,86fdf631f6102e05,e154b1f221ab36cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:31.714426+08:00, lastProcessTime=2025-10-27T11:10:21.715988+08:00 2025-10-27 11:10:31.738 INFO 2458 --- [ocp-schedules-25,9c272473ef6d1f90,d756f5ffa3bd95b5] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-10-27 11:10:31.738 INFO 2458 --- [ocp-schedules-25,9c272473ef6d1f90,d756f5ffa3bd95b5] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-10-27 11:10:31.738 INFO 2458 --- [ocp-schedules-25,9c272473ef6d1f90,d756f5ffa3bd95b5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=8770 2025-10-27 11:10:31.935 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:31.714426+08:00 2025-10-27 11:10:32.000 INFO 2458 --- [ocp-schedules-25,9c272473ef6d1f90,d756f5ffa3bd95b5] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=8770 2025-10-27 11:10:32.516 WARN 2458 --- [ocp-async-8,5402c92ee85ff637,182ef56e95ca197a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:32.516 INFO 2458 --- [ocp-async-8,99f45b53ec4e592f,cbc2d06d970f5fd9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:32.707 INFO 2458 --- [prometheus-minute-schedule-1,2e927aa0a31b724f,d1ce3e682856873e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:32.715 INFO 2458 --- [prometheus-minute-schedule-1,2e927aa0a31b724f,d1ce3e682856873e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:33.048 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:33.049 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:33.761 INFO 2458 --- [ocp-schedules-30,8302f908743ffee0,d0633e0c531db3ed] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:10:33.761 INFO 2458 --- [ocp-schedules-30,8302f908743ffee0,d0633e0c531db3ed] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:10:33.803 INFO 2458 --- [ocp-schedules-27,6612c928a6d1eb67,4acf419135d3574b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:33.806 INFO 2458 --- [ocp-schedules-27,6612c928a6d1eb67,4acf419135d3574b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:34.516 INFO 2458 --- [ocp-async-1,7473f9eb1bbc6a17,23075e865d1a96c6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:34.516 WARN 2458 --- [ocp-async-2,547a6b8b02b9d36e,82c76038062d2b81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:35.069 INFO 2458 --- [ocp-schedules-4,6d0110e2adb09eb1,5a92c40083773f7c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:35.091 WARN 2458 --- [scanner-112143425-1,6d0110e2adb09eb1,f20bad1991df594b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:35.605 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:10:35.605 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:10:35.669 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:35.669 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:35.707 INFO 2458 --- [prometheus-minute-schedule-1,72e5958e56102455,2e64fc5db050a04f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:35.710 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:35.710 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:35.710 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:35.711 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:10:35.714 INFO 2458 --- [prometheus-minute-schedule-1,72e5958e56102455,2e64fc5db050a04f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:35.724 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:35.724 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:35.725 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:35.725 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:10:35.799 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:35.800 INFO 2458 --- [ocp-schedules-9,c775241e4f7c9869,b0ba5d0196180c63] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:10:35.973 INFO 2458 --- [ocp-schedules-12,ee2b873dedce0095,1add65feb6ab0dc3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:36.049 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:36.050 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:36.516 WARN 2458 --- [ocp-async-6,7317d0d513efc9aa,e5ca80bb565956e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:36.516 INFO 2458 --- [ocp-async-9,96c244127783a9b6,0695677aac3eae69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:37.362 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:37.362 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:10:37.362 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:10:37.362 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:10:37.363 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:10:37.363 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:10:37.372 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:37.375 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:37.375 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:37.375 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:37.375 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:37.377 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:37.378 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:37.379 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:37.379 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:37.379 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:37.524 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:10:37.525 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:10:37.529 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:10:37.533 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:10:37.537 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:10:37.539 INFO 2458 --- [ocp-schedules-26,e1693434735f20e0,2612456f4126be43] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:10:37.708 INFO 2458 --- [prometheus-cache-monitor1,91794ef16a6a09e4,468c4dbab147bcb8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20311,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:10:37.708 INFO 2458 --- [prometheus-cache-monitor1,91794ef16a6a09e4,468c4dbab147bcb8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:10:38.512 INFO 2458 --- [ocp-async-3,12f4c68afbb4a2d1,406509e0a03641a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:38.516 INFO 2458 --- [ocp-async-7,a06922b2c8734498,4fdd7a45ce9a8390] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:38.516 WARN 2458 --- [ocp-async-8,1acb510a53f23cbc,d602d6c36c8d570c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:38.611 INFO 2458 --- [ocp-schedules-3,aab0a6764b615abd,8f23e408e7a50c58] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:10:38.617 INFO 2458 --- [ocp-schedules-3,aab0a6764b615abd,8f23e408e7a50c58] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:38.618 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:10:38.618 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:10:38.618 INFO 2458 --- [ocp-schedules-3,aab0a6764b615abd,8f23e408e7a50c58] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:10:38.707 INFO 2458 --- [prometheus-minute-schedule-1,ffb7e22515f4e2c7,97141c7e003a50f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:38.716 INFO 2458 --- [prometheus-minute-schedule-1,ffb7e22515f4e2c7,97141c7e003a50f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:39.050 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:39.051 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:40.015 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:10:40.015 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:10:40.061 INFO 2458 --- [Check obproxy1,0c4905a961ff0f4c,054ef1d6bb27a0fa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:10:40.061 INFO 2458 --- [Check obproxy1,0c4905a961ff0f4c,054ef1d6bb27a0fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:10:40.068 INFO 2458 --- [Collect obproxy1,22e4da9ca6906b47,4c97d6c08c078f36] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:10:40.068 INFO 2458 --- [Collect obproxy1,22e4da9ca6906b47,4c97d6c08c078f36] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:10:40.090 INFO 2458 --- [Check obproxy1,0c4905a961ff0f4c,054ef1d6bb27a0fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:10:40.094 INFO 2458 --- [Check obproxy1,0c4905a961ff0f4c,054ef1d6bb27a0fa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:10:40.094 INFO 2458 --- [Check obproxy1,0c4905a961ff0f4c,054ef1d6bb27a0fa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:10:40.099 INFO 2458 --- [Collect obproxy1,22e4da9ca6906b47,4c97d6c08c078f36] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:10:40.102 INFO 2458 --- [Collect obproxy1,22e4da9ca6906b47,4c97d6c08c078f36] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:10:40.105 INFO 2458 --- [Collect obproxy1,22e4da9ca6906b47,4c97d6c08c078f36] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:10:40.517 INFO 2458 --- [ocp-async-2,175b90dce3f6deb7,f8c51270a317ec37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:40.517 WARN 2458 --- [ocp-async-1,1f46671bc56f9657,2609b9446b364909] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:40.694 INFO 2458 --- [ocp-schedules-22,285262dc521236bf,2bae5e9d640b31fd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:10:40.694 INFO 2458 --- [ocp-schedules-22,285262dc521236bf,2bae5e9d640b31fd] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:10:40.717 INFO 2458 --- [ocp-schedules-22,285262dc521236bf,2bae5e9d640b31fd] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:10:41.707 INFO 2458 --- [prometheus-minute-schedule-1,d70e6b4c92d594b2,b5f46a9a08aade75] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:41.715 INFO 2458 --- [prometheus-minute-schedule-1,d70e6b4c92d594b2,b5f46a9a08aade75] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:41.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:41.715094+08:00, lastProcessTime=2025-10-27T11:10:31.714426+08:00 2025-10-27 11:10:41.946 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:41.715094+08:00 2025-10-27 11:10:42.051 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:42.052 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:42.517 INFO 2458 --- [ocp-async-5,507c00d3aefebf00,6f347ee70fb92c44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:42.517 WARN 2458 --- [ocp-async-9,cc9d7b9762bf0aab,57b8e11c1e692e18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:43.814 INFO 2458 --- [ocp-schedules-24,3477c5d4e6164e44,3889a6d25aade452] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:43.817 INFO 2458 --- [ocp-schedules-24,3477c5d4e6164e44,3889a6d25aade452] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:44.014 INFO 2458 --- [ocp-schedules-21,d477676e6a18837c,a00039dc485c347c] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:10:44.517 WARN 2458 --- [ocp-async-3,dfd396760845789a,5d00e5c77ee030a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:44.517 INFO 2458 --- [ocp-async-4,119c263b758f22c9,bb158416d08ff1ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:44.707 INFO 2458 --- [prometheus-minute-schedule-1,5c051a3ed1f92541,bea0b225720a8d2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:44.716 INFO 2458 --- [prometheus-minute-schedule-1,5c051a3ed1f92541,bea0b225720a8d2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:45.052 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:45.052 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:45.102 INFO 2458 --- [prometheus-second-schedule-1,c80f1ac19dce0d7f,4c7f27260d83847c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:10:45.214 INFO 2458 --- [ocp-schedules-14,1f57f39cc10f7d7b,3389c2bf2207b654] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:45.258 WARN 2458 --- [scanner-112143623-1,1f57f39cc10f7d7b,326f463e038dff31] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:45.308 INFO 2458 --- [alarm-detect-8,1f57f39cc10f7d7b,0f6f058a3f1d35de] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534575 ~ 1761534635 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534600, value=202934.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202934.56, values={1761534600=202934.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:10:45.309 INFO 2458 --- [alarm-detect-8,1f57f39cc10f7d7b,0f6f058a3f1d35de] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:10:45.349 INFO 2458 --- [alarm-detect-19,1f57f39cc10f7d7b,47f3e875e58eff21] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534575 ~ 1761534635 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534600, value=93.0)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.0, values={1761534600=93.0}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:10:45.350 INFO 2458 --- [alarm-detect-19,1f57f39cc10f7d7b,47f3e875e58eff21] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:10:45.391 INFO 2458 --- [alarm-detect-13,1f57f39cc10f7d7b,3f3db5b1d526504b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534575 ~ 1761534635 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534600, value=202935.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202935.912, values={1761534600=202935.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:10:45.392 INFO 2458 --- [alarm-detect-13,1f57f39cc10f7d7b,3f3db5b1d526504b] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:10:45.406 INFO 2458 --- [alarm-detect-3,1f57f39cc10f7d7b,8531537805692990] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534615 ~ 1761534635 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534630, value=87.072)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534630, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534630=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:10:45.980 INFO 2458 --- [ocp-schedules-19,f0181d1e9ff9851e,e20f52b8dfb2d06d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:46.067 INFO 2458 --- [ocp-schedules-16,dd22fd77090966b7,f0095672f8c1dbf2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:10:46.067 INFO 2458 --- [ocp-schedules-16,dd22fd77090966b7,f0095672f8c1dbf2] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:10:46.074 INFO 2458 --- [ocp-schedules-16,dd22fd77090966b7,f0095672f8c1dbf2] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:10:46.074 INFO 2458 --- [ocp-schedules-16,dd22fd77090966b7,f0095672f8c1dbf2] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:10:46.517 INFO 2458 --- [ocp-async-1,508fe0c5ddfbe717,22375c183db49a88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:46.517 WARN 2458 --- [ocp-async-10,5591149a908a5233,bf73127f9fe93f7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:46.835 INFO 2458 --- [ocp-schedules-3,7fce94a414e19180,fe01595566700364] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:10:46.843 INFO 2458 --- [ocp-schedules-3,7fce94a414e19180,fe01595566700364] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:46.843 INFO 2458 --- [ocp-schedules-3,7fce94a414e19180,fe01595566700364] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:10:47.419 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,95a4937e7ad064f3,a81a34b51cbffb03] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=95a4937e7ad064f3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:10:47.419 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,95a4937e7ad064f3,a81a34b51cbffb03] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=95a4937e7ad064f3, duration=1 ms] 2025-10-27 11:10:47.707 INFO 2458 --- [prometheus-minute-schedule-1,678d8149eab4a81f,90857826973ede4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:47.716 INFO 2458 --- [prometheus-minute-schedule-1,678d8149eab4a81f,90857826973ede4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:47.985 INFO 2458 --- [ocp-schedules-9,129a38fe59f8d4c6,fe507d8d13799a8f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:10:47.985+08:00, clusters=[1, 2] 2025-10-27 11:10:47.999 INFO 2458 --- [ocp-schedules-9,129a38fe59f8d4c6,fe507d8d13799a8f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:10:48.001 INFO 2458 --- [ocp-schedules-9,129a38fe59f8d4c6,fe507d8d13799a8f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:10:48.052 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:48.053 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:48.512 INFO 2458 --- [ocp-async-2,59df065fad979051,0dddde3b8ac4398c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:48.517 INFO 2458 --- [ocp-async-9,c438402cda500cac,8b50a10a1e6f0abf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:48.517 WARN 2458 --- [ocp-async-5,cd72f7142018a19e,4e25acd566490b4d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:50.517 WARN 2458 --- [ocp-async-8,f65e24a64040fdde,b137c4cfc92da15c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:50.517 INFO 2458 --- [ocp-async-4,a89ff48ac3244494,84ee49710f47f096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:50.707 INFO 2458 --- [prometheus-minute-schedule-1,fb7289593d01375b,3581c3e05a747949] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:50.717 INFO 2458 --- [prometheus-minute-schedule-1,fb7289593d01375b,3581c3e05a747949] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:51.053 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:51.054 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:51.411 INFO 2458 --- [ocp-schedules-4,f8ba42c9ba1bfe21,f91ba80426df3439] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:51.411 INFO 2458 --- [ocp-schedules-7,fb5548bddf4f6eb0,00657fed12ccf477] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:51.411 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:10:51.411 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:10:51.411 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:10:51.412 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:10:51.425 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:51.426 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:51.426 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:51.427 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:51.427 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:51.427 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:51.427 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:51.428 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:51.429 INFO 2458 --- [ocp-schedules-16,a2759f9f24792261,62a6f31673874aa4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-clockdiff1,a2759f9f24792261,204588993908285c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-clockdiff2,a2759f9f24792261,b66a39ceb72a133f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-clockdiff4,a2759f9f24792261,f2b26ae7270750f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-clockdiff3,a2759f9f24792261,5bfb22d63dd8d7ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-clockdiff5,a2759f9f24792261,ea395b86a4cf992b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:10:51.429 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:51.430 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:51.431 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:51.431 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:51.432 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:51.432 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:51.432 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:51.433 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:51.433 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:51.433 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:51.434 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:51.443 INFO 2458 --- [ocp-schedules-24,ee5e6c146a89d0cc,cbc22dd5aa3b0ac4] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:10:51.444 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:51.459 INFO 2458 --- [host-check-clockdiff5,a2759f9f24792261,ea395b86a4cf992b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534651 1 1 , error= 2025-10-27 11:10:51.472 INFO 2458 --- [host-check-clockdiff1,a2759f9f24792261,204588993908285c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534651 0 0 , error= 2025-10-27 11:10:51.478 INFO 2458 --- [host-check-clockdiff2,a2759f9f24792261,b66a39ceb72a133f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534651 -1 -1 , error= 2025-10-27 11:10:51.478 INFO 2458 --- [host-check-clockdiff4,a2759f9f24792261,f2b26ae7270750f2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534651 -1 -1 , error= 2025-10-27 11:10:51.486 INFO 2458 --- [host-check-clockdiff3,a2759f9f24792261,5bfb22d63dd8d7ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534651 0 0 , error= 2025-10-27 11:10:51.545 INFO 2458 --- [ocp-schedules-21,476a350eacd6b6c1,c29d181f0af324c7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:10:51.549 INFO 2458 --- [ocp-schedules-19,4ee7953bc7f3bca3,b6a6b3028709783c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:10:51.552 INFO 2458 --- [ocp-schedules-21,476a350eacd6b6c1,c29d181f0af324c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:51.554 INFO 2458 --- [ocp-schedules-18,5abfeb734cdd0db0,e0b13302c342c2cd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:10:51.556 INFO 2458 --- [ocp-schedules-19,4ee7953bc7f3bca3,b6a6b3028709783c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:51.558 INFO 2458 --- [ocp-schedules-18,5abfeb734cdd0db0,e0b13302c342c2cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:51.560 INFO 2458 --- [ocp-schedules-19,4ee7953bc7f3bca3,b6a6b3028709783c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:10:51.568 INFO 2458 --- [ocp-schedules-13,d5cf7ed567daeabb,0bdb34e71af8f3d8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:10:51.570 INFO 2458 --- [ocp-schedules-17,0132e67d83d398e3,7e5b4bd7d4e21639] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:10:51.571 INFO 2458 --- [ocp-schedules-28,9cc43336450674dd,c82fdc88af1a307f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:10:51.585 INFO 2458 --- [ocp-schedules-17,0132e67d83d398e3,7e5b4bd7d4e21639] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:10:51.590 INFO 2458 --- [ocp-schedules-13,d5cf7ed567daeabb,0bdb34e71af8f3d8] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:10:51.624 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:10:51.626 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:10:51.626651548+08:00 2025-10-27 11:10:51.633 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:10:51.626+08:00 2025-10-27 11:10:51.641 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:10:51.644 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.645 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:51.646 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:10:51.646 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:10:51.646 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:10:51.646 INFO 2458 --- [host-check-agent-status1,fb5548bddf4f6eb0,53b3bf5a8e9d89fc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:10:51.649 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:10:51.626+08:00 and status AVAILABLE 2025-10-27 11:10:51.656 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:10:51.661 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:10:51.662 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:10:51.662 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.662 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.663 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:10:51.663 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.663 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.663 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.663 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:10:51.664 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:51.664 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:10:51.663426463+08:00 2025-10-27 11:10:51.664 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:10:51.663894354+08:00 2025-10-27 11:10:51.665 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:10:51.664+08:00 2025-10-27 11:10:51.665 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:10:51.664+08:00 2025-10-27 11:10:51.666 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:10:51.666 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:10:51.666 INFO 2458 --- [host-check-agent-status4,fb5548bddf4f6eb0,c19198f9dc4b4a0b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:10:51.667 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:10:51.667 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:10:51.667 INFO 2458 --- [host-check-agent-status3,fb5548bddf4f6eb0,d84ea1ca2aa1c668] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:10:51.670 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:10:51.671 INFO 2458 --- [ocp-schedules-21,476a350eacd6b6c1,c29d181f0af324c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:10:51.674 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:10:51.664+08:00 and status AVAILABLE 2025-10-27 11:10:51.676 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:10:51.664+08:00 and status AVAILABLE 2025-10-27 11:10:51.677 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:10:51.678 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:10:51.677841546+08:00 2025-10-27 11:10:51.678 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:10:51.678+08:00 2025-10-27 11:10:51.679 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:10:51.681 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:10:51.689 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:10:51.678+08:00 and status AVAILABLE 2025-10-27 11:10:51.693 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:10:51.694 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.695 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:51.695 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:51.696 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:10:51.698 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:10:51.698 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:10:51.698 INFO 2458 --- [host-check-agent-status5,fb5548bddf4f6eb0,f5b5497ee5508b29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:10:51.699 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:51.702 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:10:51.703 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:10:51.705 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:10:51.706 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:10:51.709 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:10:51.713 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:10:51.713 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:10:51.713 INFO 2458 --- [host-check-agent-status2,fb5548bddf4f6eb0,46ea272d3968b8a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:10:51.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:10:51.714402+08:00, lastProcessTime=2025-10-27T11:10:41.715094+08:00 2025-10-27 11:10:51.717 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:10:51.722 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:10:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.723 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.723 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:51.724 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:10:51.724 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:10:51.723414249+08:00 2025-10-27 11:10:51.726 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:10:51.724+08:00 2025-10-27 11:10:51.739 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:10:51.724+08:00 and status AVAILABLE 2025-10-27 11:10:51.762 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:10:51.781 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:51.782 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:51.786 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:51.789 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:51.791 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:10:51.798 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:10:51.804 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:10:51.806 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:10:51.822 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:51.835 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:51.835 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.835 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.836 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.836 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.836 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.837 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:10:51.852 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:10:51.852 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:10:51.853 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:10:51.853 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:10:51.859 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:10:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:51.871 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.872 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:51.873 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.876 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:10:51.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:51.880 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:10:51.886 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:10:51.892 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:51.892 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:51.894 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:51.909 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:10:51.909 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:10:51.912 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:10:51.915 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:10:51.918 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:51.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:51.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.944 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.944 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:10:51.944 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.944 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.945 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:51.945 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:10:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.952 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:10:51.953 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:51.960 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:10:51.960 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:10:51.961 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:10:51.967 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:10:51.975 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:51.983 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:51.983 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:10:51.983 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:51.984 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:51.985 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:10:51.990 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:10:51.991 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:10:51.993 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:51.995 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:10:51.999 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:52.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.003 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:52.007 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:10:52.010 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:10:52.010 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:10:52.017 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:10:52.019 INFO 2458 --- [host-check-host-status4,f8ba42c9ba1bfe21,59da6d8191095411] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:10:52.020 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:10:51.714402+08:00 2025-10-27 11:10:52.021 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:10:52.027 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:52.049 INFO 2458 --- [host-check-host-status5,f8ba42c9ba1bfe21,5da7c210eb157870] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:10:52.057 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:10:52.061 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:10:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:52.063 INFO 2458 --- [host-check-host-status3,f8ba42c9ba1bfe21,9326834161a5b842] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:10:52.077 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:10:52.091 INFO 2458 --- [host-check-host-status1,f8ba42c9ba1bfe21,bf9bb121e26af428] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:10:52.093 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:10:52.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:52.107 INFO 2458 --- [host-check-host-status2,f8ba42c9ba1bfe21,ce41da235059db6b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:10:52.136 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:10:52.136 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.137 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:10:52.139 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.140 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:10:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:10:52.156 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:10:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:10:52.179 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:52.179 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.180 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:10:52.202 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:10:52.202 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:10:52.211 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.254 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.255 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:10:52.258 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:10:52.269 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:10:52.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:10:52.282 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:10:52.343 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:10:52.392 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:10:52.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:10:52.395 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.395 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.396 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:10:52.432 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:52.490 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:52.491 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:10:52.517 WARN 2458 --- [ocp-async-10,c44b50847e25c13e,924d896326cddf48] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:52.517 INFO 2458 --- [ocp-async-6,50841d2b7a937dbe,23f5c566455b0b5e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:52.533 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:10:52.533 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:10:52.533 INFO 2458 --- [ocp-schedules-18,5abfeb734cdd0db0,e0b13302c342c2cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:10:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:10:53.707 INFO 2458 --- [prometheus-minute-schedule-1,f8f54acf743a202d,4d8ffca525eff405] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:53.716 INFO 2458 --- [prometheus-minute-schedule-1,f8f54acf743a202d,4d8ffca525eff405] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:53.824 INFO 2458 --- [ocp-schedules-9,3fff3f983469e711,c42069426aa0219c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:10:53.828 INFO 2458 --- [ocp-schedules-9,3fff3f983469e711,c42069426aa0219c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:10:54.054 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:54.055 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:54.518 WARN 2458 --- [ocp-async-2,61e2c83b7c1baa9e,10310cd40ad8deca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:54.518 INFO 2458 --- [ocp-async-8,8d97cc27d16ed9f0,e24a6d5b1a72cc46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:54.567 INFO 2458 --- [ocp-schedules-22,6a3fada50c497a63,53cc05127d15d21d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:10:55.427 INFO 2458 --- [ocp-schedules-1,8e4913e0bd6608b4,72df205534f961da] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:10:55.471 WARN 2458 --- [scanner-112143893-1,8e4913e0bd6608b4,2c352f55df6dec62] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:10:55.567 INFO 2458 --- [alarm-detect-8,8e4913e0bd6608b4,8df40c856c9f90be] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534630 ~ 1761534645 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534645, value=63.781)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534645, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534645=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:10:55.987 INFO 2458 --- [ocp-schedules-3,6d16be6fc036788a,4ba5b463d5f81964] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:10:56.518 WARN 2458 --- [ocp-async-3,ff83e4fbb63c54c3,7898f6641318b888] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:56.518 INFO 2458 --- [ocp-async-3,0aaaae24fa2e45af,e52664ba55a5cfd5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:56.707 INFO 2458 --- [prometheus-minute-schedule-1,1d96ec08abbb71e5,f9f4234d4051555d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:56.715 INFO 2458 --- [prometheus-minute-schedule-1,1d96ec08abbb71e5,f9f4234d4051555d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:57.055 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:10:57.057 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:10:58.512 INFO 2458 --- [ocp-async-1,904c04dca490c491,bd7fbd39faa8847c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:10:58.518 WARN 2458 --- [ocp-async-10,589b0f1a509a3242,ec26ad7eb34915cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:10:58.518 INFO 2458 --- [ocp-async-5,72eb7774ffac7955,7e7676b385fc0d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:10:59.223 INFO 2458 --- [ocp-schedules-15,7fd77a973ec3ef70,36928a342a0bdb8a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:10:59.429 INFO 2458 --- [ocp-schedules-17,26e7831e9fe2997d,f0afd40da491a1aa] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:10:59.437 INFO 2458 --- [ocp-schedules-17,26e7831e9fe2997d,f0afd40da491a1aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:10:59.525 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:10:59.525 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.529 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:10:59.537 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.538 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.562 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.563 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.590 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.590 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.591 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.593 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.594 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.594 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:10:59.612 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.612 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.613 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.613 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.623 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.624 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.640 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.640 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.640 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.641 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.652 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.652 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.666 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.666 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.666 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.667 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:10:59.669 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.669 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.669 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.670 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.684 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.685 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.686 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.687 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.687 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.688 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.689 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.689 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.690 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.692 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.693 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.693 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:10:59.698 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:10:59.700 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.700 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.707 INFO 2458 --- [prometheus-minute-schedule-1,a32bbd5f5107e8a9,d4915043b5348244] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534660, interval=60 2025-10-27 11:10:59.715 INFO 2458 --- [prometheus-minute-schedule-1,a32bbd5f5107e8a9,d4915043b5348244] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:10:59.728 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.728 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.729 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.731 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:10:59.733 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.733 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.733 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.734 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:10:59.758 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:10:59.759 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.759 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:10:59.765 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.765 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.766 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.766 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:10:59.769 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:10:59.769 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:10:59.769 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:10:59.769 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:10:59.773 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:10:59.773 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:10:59.795 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.795 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.796 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.796 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:10:59.798 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.798 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.798 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.799 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:10:59.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.825 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.826 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:10:59.853 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.853 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.853 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.854 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:10:59.901 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.901 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.901 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.902 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.902 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.902 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.903 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.903 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.904 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.904 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.904 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.905 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.905 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.906 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.906 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.906 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.908 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.908 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.908 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.909 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.910 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.910 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.910 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.911 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.911 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.911 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.911 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.912 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.913 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.913 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.913 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.913 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.916 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.916 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.916 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:10:59.917 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.917 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.918 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:10:59.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:10:59.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:10:59.918 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:10:59.919 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:10:59.920 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:10:59.922 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:11:00.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:00.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:00.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:00.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:00.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:00.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:00.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:11:00.047 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:11:00.048 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:11:00.053 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:11:00.053 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:11:00.053 INFO 2458 --- [ocp-schedules-17,26e7831e9fe2997d,f0afd40da491a1aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:11:00.057 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:00.057 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:00.518 WARN 2458 --- [ocp-async-2,3c467ef728bd80a3,7163713fbdfbe5a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:00.518 INFO 2458 --- [ocp-async-9,3c844d7c49d05f57,6b006841129039e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:01.713338+08:00, lastProcessTime=2025-10-27T11:10:51.714402+08:00 2025-10-27 11:11:01.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:01.713338+08:00 2025-10-27 11:11:02.437 INFO 2458 --- [ocp-schedules-15,15c941958b402b08,9d88fa55653e1af5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:11:02.445 INFO 2458 --- [ocp-schedules-15,15c941958b402b08,9d88fa55653e1af5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:02.483 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:02.484 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:11:02.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:02.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:02.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:02.503 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:02.504 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:02.504 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:11:02.518 WARN 2458 --- [ocp-async-4,9bd01109d0022fa4,00cfcd994c48e7c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:02.518 INFO 2458 --- [ocp-async-3,ec0f995f8e16d027,153e9448b5bb7d13] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:02.518 INFO 2458 --- [ocp-schedules-15,15c941958b402b08,9d88fa55653e1af5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:11:02.708 INFO 2458 --- [prometheus-minute-schedule-1,143e9c7b56654e43,f5aab455769c4d99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:02.715 INFO 2458 --- [prometheus-minute-schedule-1,143e9c7b56654e43,f5aab455769c4d99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:03.057 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:03.059 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:03.835 INFO 2458 --- [ocp-schedules-19,2db2263a7a290969,d9ec22d915cb93fa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:03.839 INFO 2458 --- [ocp-schedules-19,2db2263a7a290969,d9ec22d915cb93fa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:04.518 INFO 2458 --- [ocp-async-5,ccc78e22487e5586,0c2e3f32c3cfcce9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:04.518 WARN 2458 --- [ocp-async-1,5470ff513b1657f6,9ba5fd2e933be828] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:05.574 INFO 2458 --- [ocp-schedules-5,83e26df58229ce08,bad9960f6df9b9fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:05.604 WARN 2458 --- [scanner-112144084-1,83e26df58229ce08,f8ece4715b356ff4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:05.732 INFO 2458 --- [prometheus-minute-schedule-1,790f4f3e66318ca5,722aca5224a79242] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:05.739 INFO 2458 --- [prometheus-minute-schedule-1,790f4f3e66318ca5,722aca5224a79242] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:05.995 INFO 2458 --- [ocp-schedules-4,be7358bdfbec40fd,a51bd97501586cf1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:06.059 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:06.060 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:06.518 WARN 2458 --- [ocp-async-5,96747cd36dc14a77,0ff620580485bd1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:06.518 INFO 2458 --- [ocp-async-2,af39f0d813205f68,5c84e5173ec62419] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:06.846 INFO 2458 --- [ocp-schedules-27,de55253745d8fd88,6c6e454453598b2b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:11:06.855 INFO 2458 --- [ocp-schedules-27,de55253745d8fd88,6c6e454453598b2b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:06.855 INFO 2458 --- [ocp-schedules-27,de55253745d8fd88,6c6e454453598b2b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:07.736 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:11:07.736+08:00, tenantSize=6 2025-10-27 11:11:07.782 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761534667736000 2025-10-27 11:11:07.782 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761534367584000 2025-10-27 11:11:07.819 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:11:07.829 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761534667824000 2025-10-27 11:11:07.829 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761534367698000 2025-10-27 11:11:07.850 INFO 2458 --- [ocp-schedules-5,82984f0901b03be8,592ea1d928020b48] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:11:08.513 INFO 2458 --- [ocp-async-2,ac45a47407b529c5,fbd7c88ca5c29adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:08.519 WARN 2458 --- [ocp-async-7,7a083bfa3c505f23,c9443ed4107a09c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:08.519 INFO 2458 --- [ocp-async-3,4f5689796006a7a7,ca08510df90f349d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:08.707 INFO 2458 --- [prometheus-minute-schedule-1,4f4dbaf2d5d3bf0e,aa575b04dd32bd28] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:08.714 INFO 2458 --- [prometheus-minute-schedule-1,4f4dbaf2d5d3bf0e,aa575b04dd32bd28] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:09.060 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:09.061 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:10.012 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:11:10.014 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:11:10.049 INFO 2458 --- [Check obproxy1,2537279bf615ea87,832495e60a050ef4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:11:10.049 INFO 2458 --- [Collect obproxy1,b48120237214d6d2,9918468af7c03d1e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:11:10.049 INFO 2458 --- [Check obproxy1,2537279bf615ea87,832495e60a050ef4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:11:10.049 INFO 2458 --- [Collect obproxy1,b48120237214d6d2,9918468af7c03d1e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:11:10.079 INFO 2458 --- [Collect obproxy1,b48120237214d6d2,9918468af7c03d1e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:11:10.080 INFO 2458 --- [Check obproxy1,2537279bf615ea87,832495e60a050ef4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:11:10.083 INFO 2458 --- [Collect obproxy1,b48120237214d6d2,9918468af7c03d1e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:11:10.084 INFO 2458 --- [Check obproxy1,2537279bf615ea87,832495e60a050ef4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:11:10.084 INFO 2458 --- [Check obproxy1,2537279bf615ea87,832495e60a050ef4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:11:10.085 INFO 2458 --- [Collect obproxy1,b48120237214d6d2,9918468af7c03d1e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:11:10.176 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:11:10.519 WARN 2458 --- [ocp-async-3,311b525eba5b34c9,98c811f24b9c365a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:10.519 INFO 2458 --- [ocp-async-3,645d051093128a65,dd7809e235a97a04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:10.546 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534600, nodeCount=94, affectRows=94 2025-10-27 11:11:10.615 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534600, nodeCount=101, affectRows=101 2025-10-27 11:11:10.634 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534600, nodeCount=101, affectRows=101 2025-10-27 11:11:11.707 INFO 2458 --- [prometheus-minute-schedule-1,2629591553dab554,1a1195a629d3a6c3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:11.714 INFO 2458 --- [prometheus-minute-schedule-1,2629591553dab554,1a1195a629d3a6c3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:11.724 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:11.719072+08:00, lastProcessTime=2025-10-27T11:11:01.713338+08:00 2025-10-27 11:11:11.931 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:11.719072+08:00 2025-10-27 11:11:12.061 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:12.061 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:12.519 WARN 2458 --- [ocp-async-3,7cb4e9911e5f9cf0,5155c01383b72b1e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:12.519 INFO 2458 --- [ocp-async-9,43d23a397a1bb35f,716087fea51f698c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:13.726 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534600, nodeCount=525, affectRows=525 2025-10-27 11:11:13.849 INFO 2458 --- [ocp-schedules-13,e5e876c83f7e9e42,0be4697f6876c7fd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:13.852 INFO 2458 --- [ocp-schedules-13,e5e876c83f7e9e42,0be4697f6876c7fd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:14.221 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=1000, affectRows=1000 2025-10-27 11:11:14.226 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=994, validCount=992 2025-10-27 11:11:14.278 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=1000, affectRows=1000 2025-10-27 11:11:14.282 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=999, validCount=995 2025-10-27 11:11:14.296 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570098) Duplicate entry '835370603008958464-1761534540' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570098) Duplicate entry '835370603008958464-1761534540' for key 'PRIMARY' 2025-10-27 11:11:14.519 WARN 2458 --- [ocp-async-4,a49c662285db2f6e,25554b92a822aff2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:14.519 INFO 2458 --- [ocp-async-2,c86b4039ff0d1f81,7b632fc935d75fd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:14.595 INFO 2458 --- [ocp-schedules-22,3dffb7ece2a16ce1,59dae07c00409616] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:11:14.595+08:00, clusters=[1, 2] 2025-10-27 11:11:14.597 INFO 2458 --- [ocp-schedules-22,3dffb7ece2a16ce1,59dae07c00409616] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:11:14.599 INFO 2458 --- [ocp-schedules-22,3dffb7ece2a16ce1,59dae07c00409616] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:11:14.707 INFO 2458 --- [prometheus-minute-schedule-1,f6e201a2f4c18578,8aebe5bb8ef03302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:14.715 INFO 2458 --- [prometheus-minute-schedule-1,f6e201a2f4c18578,8aebe5bb8ef03302] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:14.807 INFO 2458 --- [ocp-schedules-25,c89523cb195f1aee,653030848d13e3fe] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:11:14.807 INFO 2458 --- [ocp-schedules-25,c89523cb195f1aee,653030848d13e3fe] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:11:14.865 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:11:14.865 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:11:14.866 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:11:14.866 INFO 2458 --- [Collect all tenants major compactions5,c89523cb195f1aee,f9f2a734c0c29bd1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:11:14.866 INFO 2458 --- [Collect all tenants major compactions6,c89523cb195f1aee,ddebb83fbcace0e8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:11:14.866 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:11:14.898 INFO 2458 --- [Collect all tenants major compactions5,c89523cb195f1aee,f9f2a734c0c29bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.898 INFO 2458 --- [Collect all tenants major compactions6,c89523cb195f1aee,ddebb83fbcace0e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.900 INFO 2458 --- [Collect all tenants major compactions6,c89523cb195f1aee,ddebb83fbcace0e8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:11:14.900 INFO 2458 --- [Collect all tenants major compactions5,c89523cb195f1aee,f9f2a734c0c29bd1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:11:14.904 INFO 2458 --- [Collect all tenants major compactions6,c89523cb195f1aee,ddebb83fbcace0e8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:11:14.904 INFO 2458 --- [Collect all tenants major compactions5,c89523cb195f1aee,f9f2a734c0c29bd1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.914 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.915 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:14.916 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:11:14.918 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:11:14.919 INFO 2458 --- [Collect all tenants major compactions1,c89523cb195f1aee,3342d33c0948a356] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:11:14.920 INFO 2458 --- [Collect all tenants major compactions3,c89523cb195f1aee,768067a762450565] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:11:14.920 INFO 2458 --- [Collect all tenants major compactions4,c89523cb195f1aee,31cd315922a5f228] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:11:14.926 INFO 2458 --- [Collect all tenants major compactions2,c89523cb195f1aee,9b13f6f3b42843f6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:11:14.927 INFO 2458 --- [ocp-schedules-25,c89523cb195f1aee,653030848d13e3fe] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:11:15.008 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=1000, affectRows=1000 2025-10-27 11:11:15.010 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=432, affectRows=432 2025-10-27 11:11:15.012 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=431, validCount=430 2025-10-27 11:11:15.013 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=645, affectRows=645 2025-10-27 11:11:15.013 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=992, validCount=989 2025-10-27 11:11:15.016 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=645, validCount=641 2025-10-27 11:11:15.061 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:15.062 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:15.793 INFO 2458 --- [ocp-schedules-15,3f41d977e2b145cb,974b7c3806a4d12c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:15.836 WARN 2458 --- [scanner-112145865-1,3f41d977e2b145cb,a04316cd0b97541e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:16.001 INFO 2458 --- [ocp-schedules-19,6fb137e71154804f,aa90616ca52fedf9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:16.151 INFO 2458 --- [alarm-detect-9,3f41d977e2b145cb,64d3bf7d2373d806] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534645 ~ 1761534666 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534660, value=58.662)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534660, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534660=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:11:16.255 INFO 2458 --- [ocp-schedules-22,63b323311831129b,74ed3e5ed4524abb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:11:16.255 INFO 2458 --- [ocp-schedules-22,63b323311831129b,74ed3e5ed4524abb] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:11:16.263 INFO 2458 --- [ocp-schedules-22,63b323311831129b,74ed3e5ed4524abb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:11:16.263 INFO 2458 --- [ocp-schedules-22,63b323311831129b,74ed3e5ed4524abb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:11:16.519 WARN 2458 --- [ocp-async-10,c17f8ad1c09d518e,b3f8efe7d173b93f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:16.519 INFO 2458 --- [ocp-async-6,d0e2821d922eaf6e,272d06e7cea65f59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:17.585 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.595 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterV 2025-10-27 11:11:17.707 INFO 2458 --- [prometheus-minute-schedule-1,d353d5205c4039e5,915397fe5816be6e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:17.713 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.715 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.716 INFO 2458 --- [prometheus-minute-schedule-1,d353d5205c4039e5,915397fe5816be6e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:17.716 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.720 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.721 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.722 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.930 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.931 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.936 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.939 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.941 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.945 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.947 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.948 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.949 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.950 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.955 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.961 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.965 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.969 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.970 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.971 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.972 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.973 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:05:17.449, end=2025-10-27T11:10:17.579} 2025-10-27 11:11:17.974 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:11:18.062 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:18.064 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:18.130 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534600, nodeCount=1000, affectRows=1000 2025-10-27 11:11:18.133 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534600, nodeCount=1000, validCount=987 2025-10-27 11:11:18.513 INFO 2458 --- [ocp-async-5,24f354bd20cb2c04,4d633f91cf12e538] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:18.519 INFO 2458 --- [ocp-async-3,bde6ea634e7733e4,5425edabe7aba899] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:18.519 WARN 2458 --- [ocp-async-8,0d82553aeb02ce43,23e30b177aea4c7a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:19.402 INFO 2458 --- [ocp-schedules-24,2000278cbae42922,3fbf31707026f6c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:11:20.519 INFO 2458 --- [ocp-async-4,4c8cf26256aa28ca,342e50f81faaad9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:20.519 WARN 2458 --- [ocp-async-7,c94ae105810f1e10,a9f1adf6f1afb517] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:20.707 INFO 2458 --- [prometheus-minute-schedule-1,f7b3153e055735d6,3345af33e6de44f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:20.722 INFO 2458 --- [prometheus-minute-schedule-1,f7b3153e055735d6,3345af33e6de44f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:21.064 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:21.065 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:21.410 INFO 2458 --- [ocp-schedules-9,9623e9e3d1c54007,c6d66abe7f7aa60c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:21.410 INFO 2458 --- [ocp-schedules-12,e31bf84bdd227211,4ddf20c1efae9fa0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:21.410 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:11:21.410 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:11:21.411 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:11:21.423 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:21.423 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:21.424 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:21.424 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:21.424 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:21.425 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:21.426 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:21.426 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:21.428 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:21.429 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:21.429 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:21.429 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:21.429 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:21.429 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:21.430 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:21.431 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:21.431 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:21.431 INFO 2458 --- [ocp-schedules-20,53b7c5cfecb80b51,c081a89d9157c3ab] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:21.432 INFO 2458 --- [host-check-clockdiff2,53b7c5cfecb80b51,6eab8e0186b493ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:11:21.431 INFO 2458 --- [host-check-clockdiff1,53b7c5cfecb80b51,0d6cb97e282e85d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:11:21.432 INFO 2458 --- [host-check-clockdiff3,53b7c5cfecb80b51,0fa36b4884f57c26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:11:21.432 INFO 2458 --- [host-check-clockdiff4,53b7c5cfecb80b51,5d4ccf18ab05a06a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:11:21.432 INFO 2458 --- [host-check-clockdiff5,53b7c5cfecb80b51,2c8ddfd4bd5f4f3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:11:21.432 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:21.433 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:21.436 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:21.442 INFO 2458 --- [ocp-schedules-15,94f0ae9b3dd0c508,fce83dc5ca8e3a9b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:11:21.458 INFO 2458 --- [host-check-clockdiff1,53b7c5cfecb80b51,0d6cb97e282e85d8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534681 0 0 , error= 2025-10-27 11:11:21.463 INFO 2458 --- [host-check-clockdiff4,53b7c5cfecb80b51,5d4ccf18ab05a06a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534681 0 0 , error= 2025-10-27 11:11:21.465 INFO 2458 --- [host-check-clockdiff5,53b7c5cfecb80b51,2c8ddfd4bd5f4f3d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534681 0 0 , error= 2025-10-27 11:11:21.469 INFO 2458 --- [host-check-clockdiff3,53b7c5cfecb80b51,0fa36b4884f57c26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534681 -1 -1 , error= 2025-10-27 11:11:21.508 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:11:21.510 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:11:21.510 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:11:21.510 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:11:21.510 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:11:21.511 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:11:21.511 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:11:21.511 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:11:21.511197849+08:00 2025-10-27 11:11:21.512 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:11:21.511+08:00 2025-10-27 11:11:21.512 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:11:21.512469323+08:00 2025-10-27 11:11:21.512 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:11:21.511819482+08:00 2025-10-27 11:11:21.512 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:11:21.512+08:00 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:11:21.512+08:00 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:11:21.512762562+08:00 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-agent-status1,e31bf84bdd227211,e82294b8327a850c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:11:21.513 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:11:21.512631342+08:00 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:11:21.513+08:00 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-agent-status3,e31bf84bdd227211,4eb20b4009dfe951] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:11:21.514+08:00 2025-10-27 11:11:21.514 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:11:21.515 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:11:21.515 INFO 2458 --- [host-check-agent-status2,e31bf84bdd227211,fec85fa6d86dc370] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:11:21.515 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:11:21.517 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:11:21.517 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:11:21.517 INFO 2458 --- [host-check-agent-status4,e31bf84bdd227211,c1033b4b2f8ca8c9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:11:21.518 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:11:21.512+08:00 and status AVAILABLE 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-agent-status5,e31bf84bdd227211,f4366d75d6c91096] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:11:21.512+08:00 and status AVAILABLE 2025-10-27 11:11:21.521 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:11:21.511+08:00 and status AVAILABLE 2025-10-27 11:11:21.522 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:11:21.513+08:00 and status AVAILABLE 2025-10-27 11:11:21.524 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:11:21.514+08:00 and status AVAILABLE 2025-10-27 11:11:21.525 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:11:21.531 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:11:21.531 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:11:21.532 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:11:21.535 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:11:21.545 INFO 2458 --- [ocp-schedules-29,4af4ed502e8d6533,66fcdee3c59ed07f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:11:21.549 INFO 2458 --- [ocp-schedules-3,4f97753e11ec419c,813b1baa9447615b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:11:21.553 INFO 2458 --- [ocp-schedules-3,4f97753e11ec419c,813b1baa9447615b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:21.553 INFO 2458 --- [ocp-schedules-29,4af4ed502e8d6533,66fcdee3c59ed07f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:21.554 INFO 2458 --- [ocp-schedules-2,9e5412788ef56f75,b914dd701b21d1a8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:11:21.555 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:21.556 INFO 2458 --- [ocp-schedules-3,4f97753e11ec419c,813b1baa9447615b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:11:21.559 INFO 2458 --- [ocp-schedules-2,9e5412788ef56f75,b914dd701b21d1a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:21.560 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:21.567 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:21.567 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:21.568 INFO 2458 --- [ocp-schedules-14,d17c62fdf8d90b0d,145981fc776b27b2] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:11:21.574 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:21.574 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:21.577 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:21.580 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:21.581 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:21.584 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:21.614 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.614 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:11:21.644 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.645 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:21.645 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:21.655 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:11:21.655 INFO 2458 --- [ocp-schedules-29,4af4ed502e8d6533,66fcdee3c59ed07f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:11:21.656 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:21.659 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:21.660 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:11:21.660 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.661 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:21.662 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:21.667 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:21.681 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:11:21.683 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:11:21.684 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:11:21.689 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:11:21.704 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:11:21.707 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.707 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:21.710 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:11:21.712 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:11:21.716 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:11:21.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:21.716523+08:00, lastProcessTime=2025-10-27T11:11:11.719072+08:00 2025-10-27 11:11:21.723 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:11:21.750 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:21.751 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:11:21.771 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:21.773 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:21.774 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:21.777 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:21.781 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:21.786 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:11:21.793 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:11:21.794 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:11:21.797 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:11:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.803 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:11:21.806 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:11:21.821 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:11:21.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:21.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:21.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.827 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:11:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:21.842 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:11:21.844 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:11:21.851 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:11:21.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:21.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:21.859 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:11:21.871 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:21.887 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:11:21.897 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:21.899 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:21.900 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.901 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:21.902 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.905 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:11:21.907 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:21.912 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:21.912 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:21.914 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:21.920 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:21.921 INFO 2458 --- [host-check-host-status4,9623e9e3d1c54007,cc7ac0b7de20c157] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:11:21.928 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:21.929 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:21.930 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:21.931 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:11:21.931 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:11:21.935 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:11:21.940 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:11:21.940 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:11:21.945 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:11:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:21.957 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:21.960 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:21.965 INFO 2458 --- [host-check-host-status5,9623e9e3d1c54007,a98d9ab800a8481f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:11:21.965 INFO 2458 --- [host-check-host-status2,9623e9e3d1c54007,554177affbcbbd16] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:11:21.971 INFO 2458 --- [host-check-host-status3,9623e9e3d1c54007,f1a08909c8d2c3e8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:11:21.975 INFO 2458 --- [host-check-host-status1,9623e9e3d1c54007,370d827ab7f054cb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:11:21.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:21.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:22.017 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:11:22.034 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:22.034 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:22.059 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:11:22.059 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.060 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:11:22.062 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.063 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:11:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:22.082 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:11:22.106 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:22.106 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.107 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:11:22.109 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:22.109 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:22.130 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:11:22.130 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534084668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:11:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:22.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:22.184 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.185 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:22.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:11:22.198 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:11:22.207 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:22.210 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:22.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:11:22.307 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:21.716523+08:00 2025-10-27 11:11:22.334 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:11:22.334 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.335 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:11:22.340 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.341 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:11:22.381 INFO 2458 --- [host-check-clockdiff2,53b7c5cfecb80b51,6eab8e0186b493ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534682 -1 -1 , error= 2025-10-27 11:11:22.386 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:11:22.427 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:22.428 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:22.428 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:11:22.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:11:22.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:11:22.479 INFO 2458 --- [ocp-schedules-2,9e5412788ef56f75,b914dd701b21d1a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:11:22.520 WARN 2458 --- [ocp-async-10,0b03e4fcc12cc159,736e6d234282cc45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:22.520 INFO 2458 --- [ocp-async-1,9bfb5f2aa263a43b,3d6f5993b3c35e1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:23.707 INFO 2458 --- [prometheus-minute-schedule-1,dee07eb15cfa207c,24b2894dd29a6542] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:23.718 INFO 2458 --- [prometheus-minute-schedule-1,dee07eb15cfa207c,24b2894dd29a6542] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:23.860 INFO 2458 --- [ocp-schedules-9,831a5a466f6a6854,9d183f3a57a4cfac] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:23.863 INFO 2458 --- [ocp-schedules-9,831a5a466f6a6854,9d183f3a57a4cfac] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:24.065 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:24.066 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:24.520 WARN 2458 --- [ocp-async-8,88e7ac04cc94f7cc,5e5a9b50e23bf772] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:24.520 INFO 2458 --- [ocp-async-5,a3d40cbc89632162,c44d27b667fbadf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:24.667 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4858b4afc404417b,a3dd1df946e3ac93] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&version=2&ObClusterId=1760510188, client=10.40.66.107, traceId=4858b4afc404417b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase2,1760510188,2,{"ObRegion":"myoceanbase2","ObCluster":"myoceanbase2","ObRegionId":1760510188,"ObClusterId":1760510188,"Type":"PRIMARY","timestamp":1761534684666512,"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:11:24.677 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4858b4afc404417b,a3dd1df946e3ac93] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase2, obClusterId=1760510188, rootServerJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, groupName=myoceanbase2, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:11:24.677 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4858b4afc404417b,a3dd1df946e3ac93] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:11:24.677 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4858b4afc404417b,a3dd1df946e3ac93] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase2, clusterId=1760510188, affectRows=1 2025-10-27 11:11:24.678 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4858b4afc404417b,a3dd1df946e3ac93] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.66.107, traceId=4858b4afc404417b, duration=11 ms] 2025-10-27 11:11:26.010 INFO 2458 --- [ocp-schedules-18,b7dd2c715437cbf4,1cc792eb1dece279] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:26.158 INFO 2458 --- [ocp-schedules-24,76059a271714954f,a472ee5c55b44164] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:26.191 WARN 2458 --- [scanner-112146034-1,76059a271714954f,893327a757058cb0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:26.336 INFO 2458 --- [alarm-detect-16,76059a271714954f,3765579842ace288] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534660 ~ 1761534676 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534675, value=61.708)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534675, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534675=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:11:26.341 INFO 2458 --- [alarm-detect-16,76059a271714954f,3765579842ace288] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [279] will be resolved. 2025-10-27 11:11:26.341 INFO 2458 --- [alarm-detect-16,76059a271714954f,3765579842ace288] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [279] will be resolved. 2025-10-27 11:11:26.387 INFO 2458 --- [alarm-detect-16,76059a271714954f,3765579842ace288] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2025-10-27 11:11:26.520 WARN 2458 --- [ocp-async-3,2ee157598544e315,90a9a09773ba478d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:26.520 INFO 2458 --- [ocp-async-7,f2b6f424f790b7d7,4588b518b6964c1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:26.707 INFO 2458 --- [prometheus-minute-schedule-1,b0e0fb1cfcfa7089,f385dc625316c27a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:26.717 INFO 2458 --- [prometheus-minute-schedule-1,b0e0fb1cfcfa7089,f385dc625316c27a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:26.858 INFO 2458 --- [ocp-schedules-11,036a7c1288f6ac12,28679871500d426c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:11:26.866 INFO 2458 --- [ocp-schedules-11,036a7c1288f6ac12,28679871500d426c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:26.866 INFO 2458 --- [ocp-schedules-11,036a7c1288f6ac12,28679871500d426c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:27.066 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:27.067 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:28.120 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:11:28.120 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:11:28.124 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:11:28.124 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:11:28.124 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:11:28.124 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:11:28.128 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:11:28.128 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:11:28.128 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:11:28.128 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:11:28.153 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:11:28.158 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:11:28.158 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:11:28.158 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:11:28.158 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:11:28.158 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:11:28.165 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:11:28.513 INFO 2458 --- [ocp-async-4,4133efad08c40a83,3fef318683f0c584] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:28.520 WARN 2458 --- [ocp-async-10,40725b5597b85a07,22f6f9f752d49654] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:28.520 INFO 2458 --- [ocp-async-6,1d0c40603b667ff0,16adf6cd74409986] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:28.527 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:11:28.815 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6996a09c586ed356,821680cac5f2bf99] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=6996a09c586ed356, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:11:28.816 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6996a09c586ed356,821680cac5f2bf99] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=6996a09c586ed356, duration=1 ms] 2025-10-27 11:11:28.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,045623444d23559d,e2f3dc685bf8897e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest, client=10.40.67.22, traceId=045623444d23559d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,false,] 2025-10-27 11:11:28.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,045623444d23559d,e2f3dc685bf8897e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=045623444d23559d, duration=1 ms] 2025-10-27 11:11:28.871 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:11:28.871 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:11:28.873 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:11:28.873 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:11:28.873 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:11:28.873 INFO 2458 --- [ocp-schedules-6,3bb697b2bce76f02,112a6206701222a0] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:11:28.909 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.781369479S 2025-10-27 11:11:28.909 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:11:28.977 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:11:28.977 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:11:29.025 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:11:29.075 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:11:29.083 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:11:29.202 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] o.s.boot.SpringApplication : Started application in 1.078 seconds (JVM running for 257157.629) 2025-10-27 11:11:29.202 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.078515622S 2025-10-27 11:11:29.342 INFO 2458 --- [ocp-schedules-28,0c2b784dbb914314,832ed5c82f38f6e2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:11:29.350 INFO 2458 --- [ocp-schedules-28,0c2b784dbb914314,832ed5c82f38f6e2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:11:29.350 INFO 2458 --- [ocp-schedules-28,0c2b784dbb914314,832ed5c82f38f6e2] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:11:29.538 INFO 2458 --- [ocp-schedules-27,9bd5ee82c987ca5e,01aab290ddd7e024] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:11:29.707 INFO 2458 --- [prometheus-minute-schedule-1,3278b8e005bee82b,41589194af7b984d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:29.715 INFO 2458 --- [prometheus-minute-schedule-1,3278b8e005bee82b,41589194af7b984d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:30.067 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:30.067 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:30.520 INFO 2458 --- [ocp-async-8,f6bedd37d8b256a3,e50534cca1259f5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:30.520 WARN 2458 --- [ocp-async-5,b13e4de4578926ad,cf2a0b35ea3bc898] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:31.713793+08:00, lastProcessTime=2025-10-27T11:11:21.716523+08:00 2025-10-27 11:11:31.942 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:31.713793+08:00 2025-10-27 11:11:32.520 WARN 2458 --- [ocp-async-2,be19180e77b908f1,49f816b7843c7b70] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:32.520 INFO 2458 --- [ocp-async-2,ac267c509f4851c4,15b60c3de6d66f9f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:32.707 INFO 2458 --- [prometheus-minute-schedule-1,970739f0924ef14a,a7e20c4b94979dc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:32.716 INFO 2458 --- [prometheus-minute-schedule-1,970739f0924ef14a,a7e20c4b94979dc9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:33.068 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:33.068 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:33.770 INFO 2458 --- [ocp-schedules-6,8614020506464b92,7ba73c922994f6f2] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:11:33.770 INFO 2458 --- [ocp-schedules-6,8614020506464b92,7ba73c922994f6f2] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:11:33.871 INFO 2458 --- [ocp-schedules-12,7e33747ea88842ff,a56b36d25a2eca65] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:33.875 INFO 2458 --- [ocp-schedules-12,7e33747ea88842ff,a56b36d25a2eca65] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:34.520 WARN 2458 --- [ocp-async-1,027dcedd2c854f5f,1aa2da7f476ceda0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:34.521 INFO 2458 --- [ocp-async-10,742a425b58c882f6,18aae2d1e4b4959d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:35.707 INFO 2458 --- [prometheus-minute-schedule-1,9c570424661dbd67,2ab90763eeddf536] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:35.715 INFO 2458 --- [prometheus-minute-schedule-1,9c570424661dbd67,2ab90763eeddf536] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:35.806 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:11:35.806 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:11:35.859 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=root 2025-10-27 11:11:35.859 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:35.859 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:35.923 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:35.923 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:35.923 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:35.924 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:11:35.937 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:35.937 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:35.937 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:35.938 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:11:35.972 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:11:36.002 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:36.003 INFO 2458 --- [ocp-schedules-30,027bb183cb4f1a3e,6c03ea6363014c1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:11:36.009 INFO 2458 --- [ocp-schedules-18,5edccd25545d347c,41a863925b324947] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-10-27 11:11:36.014 INFO 2458 --- [ocp-schedules-18,5edccd25545d347c,41a863925b324947] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:36.030 INFO 2458 --- [ocp-schedules-22,5a0acebfaa17d862,7ac201441a6f45ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:36.068 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:36.069 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:36.087 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:36.092 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 02:04:47.657729, 2025-10-27 11:11:36.04, 1000] 2025-10-27 11:11:36.092 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2025-10-27 02:04:47.657729, endTime 2025-10-27 11:11:36.04 2025-10-27 11:11:36.113 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:36.113 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:36.114 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:36.114 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:36.114 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:36.118 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091391, 2025-10-27 11:11:36.045, 1000] 2025-10-27 11:11:36.119 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2025-10-27 08:00:44.091391, endTime 2025-10-27 11:11:36.045 2025-10-27 11:11:36.134 INFO 2458 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:36.138 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:36.160 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-17 15:45:19.719943, 2025-10-27 11:11:36.111, 1000] 2025-10-27 11:11:36.161 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2025-10-17 15:45:19.719943, endTime 2025-10-27 11:11:36.111 2025-10-27 11:11:36.196 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:36.196 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:36.196 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:36.197 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:36.197 INFO 2458 --- [Sync system event1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:36.197 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:36.219 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091611, 2025-10-27 11:11:36.156, 1000] 2025-10-27 11:11:36.220 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2025-10-27 08:00:44.091611, endTime 2025-10-27 11:11:36.156 2025-10-27 11:11:36.220 INFO 2458 --- [ocp-schedules-18,5edccd25545d347c,41a863925b324947] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=2, successCount=2 2025-10-27 11:11:36.399 INFO 2458 --- [ocp-schedules-6,521a7ac18328bb63,47569ee4719871b1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:36.426 WARN 2458 --- [scanner-112146287-1,521a7ac18328bb63,90d976dbb7eee73e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:36.521 WARN 2458 --- [ocp-async-6,525b72383683e4d6,98963ec1e52d04d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:36.521 INFO 2458 --- [ocp-async-9,f8e5f5fff8a9279e,acf8dc2373da3fe9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:37.552 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:37.552 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:11:37.552 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:11:37.552 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:11:37.552 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:11:37.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:11:37.564 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:37.565 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:37.566 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:37.566 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:37.566 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:37.569 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:37.570 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:37.570 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:37.570 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:37.570 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:37.639 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:11:37.641 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:11:37.648 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:11:37.650 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:11:37.650 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:11:37.652 INFO 2458 --- [ocp-schedules-1,cfac3c4a2e53c868,1c917f53eec694d1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:11:37.708 INFO 2458 --- [prometheus-cache-monitor1,bf1bf2242ff4a550,6403ed36dc80b445] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20308,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5076,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:11:37.708 INFO 2458 --- [prometheus-cache-monitor1,bf1bf2242ff4a550,6403ed36dc80b445] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:11:38.513 INFO 2458 --- [ocp-async-8,274c604623ebb042,d84991131063a486] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:38.521 WARN 2458 --- [ocp-async-7,e5f106b691d4dfae,cf1015f98ad94bf7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:38.521 INFO 2458 --- [ocp-async-2,d9aecc61751578d0,bec700035d5ce74b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:38.618 INFO 2458 --- [ocp-schedules-22,df59079abc6a9d18,74ce42fc7db37812] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:11:38.625 INFO 2458 --- [ocp-schedules-22,df59079abc6a9d18,74ce42fc7db37812] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:38.626 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:11:38.626 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:11:38.626 INFO 2458 --- [ocp-schedules-22,df59079abc6a9d18,74ce42fc7db37812] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:11:38.707 INFO 2458 --- [prometheus-minute-schedule-1,790652ae1af74b27,fb155f8df9924077] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:38.715 INFO 2458 --- [prometheus-minute-schedule-1,790652ae1af74b27,fb155f8df9924077] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:39.069 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:39.070 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:40.014 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:11:40.014 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:11:40.048 INFO 2458 --- [Check obproxy1,23c4bda20af52419,14d284994bf0c66b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:11:40.052 INFO 2458 --- [Check obproxy1,23c4bda20af52419,14d284994bf0c66b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:11:40.054 INFO 2458 --- [Collect obproxy1,b5ff45e6211ca597,ffb66e65aa7b2c6d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:11:40.054 INFO 2458 --- [Collect obproxy1,b5ff45e6211ca597,ffb66e65aa7b2c6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:11:40.082 INFO 2458 --- [Check obproxy1,23c4bda20af52419,14d284994bf0c66b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:11:40.085 INFO 2458 --- [Collect obproxy1,b5ff45e6211ca597,ffb66e65aa7b2c6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:11:40.085 INFO 2458 --- [Check obproxy1,23c4bda20af52419,14d284994bf0c66b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:11:40.085 INFO 2458 --- [Check obproxy1,23c4bda20af52419,14d284994bf0c66b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:11:40.088 INFO 2458 --- [Collect obproxy1,b5ff45e6211ca597,ffb66e65aa7b2c6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:11:40.091 INFO 2458 --- [Collect obproxy1,b5ff45e6211ca597,ffb66e65aa7b2c6d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:11:40.521 WARN 2458 --- [ocp-async-4,8e2ed4be53b167b6,8085aa413caba149] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:40.521 INFO 2458 --- [ocp-async-10,3c895fbcf823ea5f,90c759a5c5b66820] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:40.764 INFO 2458 --- [ocp-schedules-21,5fec039a83ddc0d5,3e510f7a2ae330c1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:11:40.765 INFO 2458 --- [ocp-schedules-21,5fec039a83ddc0d5,3e510f7a2ae330c1] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:11:40.788 INFO 2458 --- [ocp-schedules-21,5fec039a83ddc0d5,3e510f7a2ae330c1] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:11:41.707 INFO 2458 --- [prometheus-minute-schedule-1,bf46c4edca28c242,c5cdbada5b3e6540] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:41.715 INFO 2458 --- [prometheus-minute-schedule-1,bf46c4edca28c242,c5cdbada5b3e6540] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:41.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:41.714646+08:00, lastProcessTime=2025-10-27T11:11:31.713793+08:00 2025-10-27 11:11:41.933 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:41.714646+08:00 2025-10-27 11:11:42.070 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:42.071 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:42.521 WARN 2458 --- [ocp-async-6,ac39a80144e40d38,475313060b5841ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:42.521 INFO 2458 --- [ocp-async-9,249c172f060eb3ba,1d0b43bcf3dd143b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:43.886 INFO 2458 --- [ocp-schedules-28,310b6673f0f296ad,00bd2dcfa7b2fb56] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:43.890 INFO 2458 --- [ocp-schedules-28,310b6673f0f296ad,00bd2dcfa7b2fb56] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:44.026 INFO 2458 --- [ocp-schedules-27,304eb7ac710598e9,d90da95e033a7703] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:11:44.521 WARN 2458 --- [ocp-async-8,fdbd8a1c9672df7c,5ae236efdda8dfb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:44.521 INFO 2458 --- [ocp-async-9,42d99d428c27b446,7c0da7cdb2a6aebe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:44.707 INFO 2458 --- [prometheus-minute-schedule-1,69b4e1ed194962ab,7b1e4b4c3e8ad72a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:44.716 INFO 2458 --- [prometheus-minute-schedule-1,69b4e1ed194962ab,7b1e4b4c3e8ad72a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:45.071 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:45.072 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:45.102 INFO 2458 --- [prometheus-second-schedule-1,0631c8eb76c4e8e5,fb8091d2aed23895] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:11:45.778 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,bfc328491bf480a4,d36dd1c81200ef1b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=bfc328491bf480a4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:11:45.778 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,bfc328491bf480a4,d36dd1c81200ef1b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=bfc328491bf480a4, duration=2 ms] 2025-10-27 11:11:46.037 INFO 2458 --- [ocp-schedules-21,a604e08e8bf5cb94,5de22fb1fef82086] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:46.443 INFO 2458 --- [ocp-schedules-9,4b7102f6f2cf8019,a143f4fd22b659c1] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:11:46.443 INFO 2458 --- [ocp-schedules-9,4b7102f6f2cf8019,a143f4fd22b659c1] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:11:46.450 INFO 2458 --- [ocp-schedules-9,4b7102f6f2cf8019,a143f4fd22b659c1] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:11:46.450 INFO 2458 --- [ocp-schedules-9,4b7102f6f2cf8019,a143f4fd22b659c1] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:11:46.521 WARN 2458 --- [ocp-async-4,4da0818d0f7a38a9,25dea692b2fedbc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:46.521 INFO 2458 --- [ocp-async-1,f5a238b6270fcc51,ab75c7cac3be1c00] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:46.543 INFO 2458 --- [ocp-schedules-4,3b752fe16e997f9c,777edb8bac40fd21] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:46.676 WARN 2458 --- [scanner-112146489-1,3b752fe16e997f9c,a45c383460c96442] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:46.810 INFO 2458 --- [alarm-detect-10,3b752fe16e997f9c,2384c7ab06c59e97] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534636 ~ 1761534696 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534660, value=202994.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=202994.56, values={1761534660=202994.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:11:46.858 INFO 2458 --- [alarm-detect-6,3b752fe16e997f9c,87b8e496b3564105] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534636 ~ 1761534696 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534660, value=93.009)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.009, values={1761534660=93.009}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:11:46.860 INFO 2458 --- [alarm-detect-4,3b752fe16e997f9c,d08232135f679d65] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534636 ~ 1761534696 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534660, value=202995.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=202995.912, values={1761534660=202995.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:11:46.868 INFO 2458 --- [ocp-schedules-12,94f7184911dbf835,62712dc64f02af8f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:11:46.877 INFO 2458 --- [ocp-schedules-12,94f7184911dbf835,62712dc64f02af8f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:46.877 INFO 2458 --- [ocp-schedules-12,94f7184911dbf835,62712dc64f02af8f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:11:46.886 INFO 2458 --- [alarm-detect-17,3b752fe16e997f9c,7d801ee03b8ab2be] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534675 ~ 1761534696 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534690, value=62.107)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534690, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534690=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:11:46.888 INFO 2458 --- [alarm-detect-17,3b752fe16e997f9c,7d801ee03b8ab2be] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3, currentState=Firing 2025-10-27 11:11:46.888 INFO 2458 --- [alarm-detect-17,3b752fe16e997f9c,7d801ee03b8ab2be] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:11:46.908 INFO 2458 --- [alarm-detect-17,3b752fe16e997f9c,7d801ee03b8ab2be] c.o.o.alarm.service.OcpAlarmServiceImpl : delete alarm event 279 2025-10-27 11:11:46.911 INFO 2458 --- [alarm-detect-17,3b752fe16e997f9c,7d801ee03b8ab2be] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=352 2025-10-27 11:11:47.707 INFO 2458 --- [prometheus-minute-schedule-1,28bf70089c2f8ccd,ff84908d9207e3a8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:47.715 INFO 2458 --- [prometheus-minute-schedule-1,28bf70089c2f8ccd,ff84908d9207e3a8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:48.059 INFO 2458 --- [ocp-schedules-23,cb559833c466518e,61890ac3cc924c98] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:11:48.059+08:00, clusters=[1, 2] 2025-10-27 11:11:48.072 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:48.073 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:48.090 INFO 2458 --- [ocp-schedules-23,cb559833c466518e,61890ac3cc924c98] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:11:48.093 INFO 2458 --- [ocp-schedules-23,cb559833c466518e,61890ac3cc924c98] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:11:48.513 INFO 2458 --- [ocp-async-10,a383a7961d49288e,a54134809f8f7055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:48.522 WARN 2458 --- [ocp-async-5,78be5e2c4512662f,d33f237c1d029d0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:48.522 INFO 2458 --- [ocp-async-7,7ae2e458fb06b483,8e53795ac7310baf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:50.522 WARN 2458 --- [ocp-async-3,35bb0a85cd910a7f,4bca0ef620ee1068] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:50.522 INFO 2458 --- [ocp-async-3,f1259ac34b4090dd,f4180ef8fd34e666] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:50.707 INFO 2458 --- [prometheus-minute-schedule-1,5d2c0f17dbfc7044,666bca0b75e8de6c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:50.715 INFO 2458 --- [prometheus-minute-schedule-1,5d2c0f17dbfc7044,666bca0b75e8de6c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:51.073 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:51.074 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:51.410 INFO 2458 --- [ocp-schedules-14,cc65d7f8c0259301,dd89a54ae4c33e26] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:51.410 INFO 2458 --- [ocp-schedules-29,0946b2bfec7b6b54,1ca57974ad642de1] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:51.410 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:11:51.410 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:11:51.410 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:11:51.410 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:11:51.411 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:11:51.424 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:51.424 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:51.425 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:51.425 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:51.425 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:51.425 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:51.425 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:51.427 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:51.429 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:51.430 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:51.430 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:51.430 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:51.430 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:51.431 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:51.431 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:51.431 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:51.431 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:51.431 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:51.433 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:51.433 INFO 2458 --- [ocp-schedules-21,66c7c323bfdcce8d,c6d02bcdbab5cfce] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:11:51.433 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:51.434 INFO 2458 --- [host-check-clockdiff2,66c7c323bfdcce8d,dc4330a97c507990] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:11:51.434 INFO 2458 --- [host-check-clockdiff1,66c7c323bfdcce8d,04dfe6dd1196be5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:11:51.435 INFO 2458 --- [host-check-clockdiff3,66c7c323bfdcce8d,e2e84d127d3c9538] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:11:51.436 INFO 2458 --- [host-check-clockdiff4,66c7c323bfdcce8d,a49f1c6bcd97c8fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:11:51.438 INFO 2458 --- [host-check-clockdiff5,66c7c323bfdcce8d,b2905ab215a4dfa3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:11:51.444 INFO 2458 --- [ocp-schedules-16,863810261f8ecaa8,090ea7fbecfa67bf] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:11:51.454 INFO 2458 --- [host-check-clockdiff3,66c7c323bfdcce8d,e2e84d127d3c9538] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534711 0 0 , error= 2025-10-27 11:11:51.454 INFO 2458 --- [host-check-clockdiff5,66c7c323bfdcce8d,b2905ab215a4dfa3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534711 0 0 , error= 2025-10-27 11:11:51.456 INFO 2458 --- [host-check-clockdiff2,66c7c323bfdcce8d,dc4330a97c507990] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534711 -1 -1 , error= 2025-10-27 11:11:51.459 INFO 2458 --- [host-check-clockdiff1,66c7c323bfdcce8d,04dfe6dd1196be5a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534711 0 0 , error= 2025-10-27 11:11:51.460 INFO 2458 --- [host-check-clockdiff4,66c7c323bfdcce8d,a49f1c6bcd97c8fc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534711 0 0 , error= 2025-10-27 11:11:51.526 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:11:51.526 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:11:51.526 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:11:51.528 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:11:51.528 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:11:51.528547206+08:00 2025-10-27 11:11:51.529 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:11:51.529 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:11:51.528+08:00 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status4,0946b2bfec7b6b54,f711daa595bcd524] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status3,0946b2bfec7b6b54,04af9cd5c427e487] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:11:51.531 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:11:51.532 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:11:51.532 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:11:51.532 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:11:51.531939370+08:00 2025-10-27 11:11:51.532 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:11:51.531466303+08:00 2025-10-27 11:11:51.532 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:11:51.532+08:00 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:11:51.532+08:00 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:11:51.532535788+08:00 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-agent-status5,0946b2bfec7b6b54,071faa11505e913d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:11:51.533 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:11:51.533+08:00 2025-10-27 11:11:51.534 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:11:51.534 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:11:51.534 INFO 2458 --- [host-check-agent-status2,0946b2bfec7b6b54,bc5349f9ae050634] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:11:51.538 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:11:51.540 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:11:51.539972761+08:00 2025-10-27 11:11:51.540 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:11:51.540+08:00 2025-10-27 11:11:51.542 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:11:51.542 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:11:51.542 INFO 2458 --- [host-check-agent-status1,0946b2bfec7b6b54,143bc7a09369bc70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:11:51.545 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:11:51.528+08:00 and status AVAILABLE 2025-10-27 11:11:51.546 INFO 2458 --- [ocp-schedules-6,46e1ea724ba502c6,411149ab237fd8df] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:11:51.547 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:11:51.533+08:00 and status AVAILABLE 2025-10-27 11:11:51.549 INFO 2458 --- [ocp-schedules-5,e487f9c163f911c2,fed14984296e03fe] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:11:51.551 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:11:51.551 INFO 2458 --- [ocp-schedules-6,46e1ea724ba502c6,411149ab237fd8df] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:51.551 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:11:51.532+08:00 and status AVAILABLE 2025-10-27 11:11:51.551 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:11:51.540+08:00 and status AVAILABLE 2025-10-27 11:11:51.552 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:11:51.532+08:00 and status AVAILABLE 2025-10-27 11:11:51.553 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:11:51.554 INFO 2458 --- [ocp-schedules-18,4ddd35991683771a,42cebfb0b83f1348] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:11:51.554 INFO 2458 --- [ocp-schedules-5,e487f9c163f911c2,fed14984296e03fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:51.554 INFO 2458 --- [ocp-schedules-5,e487f9c163f911c2,fed14984296e03fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:11:51.556 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:11:51.557 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:11:51.557 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:11:51.558 INFO 2458 --- [ocp-schedules-18,4ddd35991683771a,42cebfb0b83f1348] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:51.566 INFO 2458 --- [ocp-schedules-8,b8ac0849d4c31f2e,acf9f86b1c12db0d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:11:51.568 INFO 2458 --- [ocp-schedules-25,c36002239b5b0a53,43295bcd089541b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:11:51.569 INFO 2458 --- [ocp-schedules-11,e065f9ca4628a0a5,3bfcf7176c33b85e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:11:51.578 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:51.579 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:11:51.581 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:51.581 INFO 2458 --- [ocp-schedules-11,e065f9ca4628a0a5,3bfcf7176c33b85e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:11:51.583 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:11:51.583 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:51.583 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:11:51.583 INFO 2458 --- [ocp-schedules-25,c36002239b5b0a53,43295bcd089541b1] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:11:51.586 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:51.586 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:11:51.590 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:11:51.592 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.593 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:11:51.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:51.594 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:11:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:51.646 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:11:51.646 INFO 2458 --- [ocp-schedules-6,46e1ea724ba502c6,411149ab237fd8df] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:11:51.656 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.656 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:51.666 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:51.668 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:51.668 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:51.671 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:51.676 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:11:51.683 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:11:51.683 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:11:51.693 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:11:51.697 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:11:51.700 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:11:51.712 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.712 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.712 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.712 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.712 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.713 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:11:51.713752+08:00, lastProcessTime=2025-10-27T11:11:41.714646+08:00 2025-10-27 11:11:51.723 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:51.725 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:11:51.727 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:11:51.729 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.730 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:11:51.735 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:11:51.744 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:11:51.746 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:11:51.771 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:51.773 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:51.776 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:51.776 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:51.779 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:11:51.784 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:51.785 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.785 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:51.790 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.791 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:11:51.791 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:11:51.791 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:11:51.793 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:11:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:51.796 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:11:51.809 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:51.809 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.809 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.809 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:11:51.810 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.810 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.810 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.811 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:11:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:51.827 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:11:51.829 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:11:51.829 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:11:51.842 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:11:51.842 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:11:51.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:51.855 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:51.859 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:51.861 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:51.869 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:51.870 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:11:51.872 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:11:51.876 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:11:51.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:51.878 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.879 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:51.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:11:51.882 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:11:51.889 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:11:51.890 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:11:51.898 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.900 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.901 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:11:51.902 INFO 2458 --- [host-check-host-status4,cc65d7f8c0259301,2f4ea5529b0c956e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:11:51.902 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:51.902 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:51.903 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:11:51.906 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:11:51.907 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:51.910 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:11:51.910 INFO 2458 --- [host-check-host-status2,cc65d7f8c0259301,b1121c9c09d3ed0e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:11:51.913 INFO 2458 --- [host-check-host-status5,cc65d7f8c0259301,36ddf31532817253] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:11:51.925 INFO 2458 --- [host-check-host-status3,cc65d7f8c0259301,58dc3d8e3597a152] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:11:51.940 INFO 2458 --- [host-check-host-status1,cc65d7f8c0259301,ea3c6d82a96ef2fa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:11:51.972 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:11:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:51.999 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:11:51.713752+08:00 2025-10-27 11:11:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:52.023 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:11:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:11:52.029 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.030 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:11:52.046 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:11:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:52.069 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:52.074 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.074 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:11:52.090 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:52.091 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:11:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:11:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:11:52.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:52.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:11:52.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:11:52.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:52.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:11:52.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:11:52.212 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:11:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:11:52.226 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:11:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:11:52.326 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:11:52.327 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.327 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.327 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.327 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:11:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.330 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:11:52.364 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:11:52.406 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:11:52.407 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:11:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:11:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:11:52.446 INFO 2458 --- [ocp-schedules-18,4ddd35991683771a,42cebfb0b83f1348] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:11:52.522 WARN 2458 --- [ocp-async-1,c9ed0911aef6263a,03f097f3ab82d3c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:52.522 INFO 2458 --- [ocp-async-5,dde841a0e18061fb,45bf8c6f92dfbeb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:11:53.221 INFO 2458 --- [ocp-schedules-25,f0e17e5c9e5d0bdd,57d3c22444112d16] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20308,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5076,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:11:53.221 INFO 2458 --- [ocp-schedules-25,f0e17e5c9e5d0bdd,57d3c22444112d16] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:11:53.707 INFO 2458 --- [prometheus-minute-schedule-1,af46ee5945b534a9,affcad4d43bf42f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:53.715 INFO 2458 --- [prometheus-minute-schedule-1,af46ee5945b534a9,affcad4d43bf42f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:53.902 INFO 2458 --- [ocp-schedules-22,dd929809452a09d6,54bfcb551b2a9d16] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:11:53.906 INFO 2458 --- [ocp-schedules-22,dd929809452a09d6,54bfcb551b2a9d16] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:11:54.074 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:54.075 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:54.522 INFO 2458 --- [ocp-async-5,5da01a27f2bfb9c7,9baa1244a91bb109] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:54.522 WARN 2458 --- [ocp-async-7,71692b824eb7b728,4b72c40caeb7ffc7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:54.614 INFO 2458 --- [ocp-schedules-7,117553af60d73f85,b754faf658f4176d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:11:56.050 INFO 2458 --- [ocp-schedules-1,425cd5a36a9b45e4,12b1d004319e8872] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:11:56.522 WARN 2458 --- [ocp-async-3,01a8e14527d33b57,81c8cc17b5959dff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:56.522 INFO 2458 --- [ocp-async-8,5ee3f1d05ff7b1da,67f6dd9a8d54de20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:56.707 INFO 2458 --- [prometheus-minute-schedule-1,3a96ff298739e105,ffe3127e78cdf8ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:56.714 INFO 2458 --- [prometheus-minute-schedule-1,3a96ff298739e105,ffe3127e78cdf8ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:11:56.931 INFO 2458 --- [ocp-schedules-3,e49cea1c79f27fe7,0f1f449933fa2dfc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:11:56.967 WARN 2458 --- [scanner-112146773-1,e49cea1c79f27fe7,b088866b8ad65e52] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:11:57.059 INFO 2458 --- [alarm-detect-12,e49cea1c79f27fe7,71217b1d71402a8e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534690 ~ 1761534707 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534705, value=63.51)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534705, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534705=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:11:57.075 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:11:57.075 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:11:57.787 INFO 2458 --- [ocp-schedules-17,ee4caec197cc8622,9254c763523138b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:11:57.794 INFO 2458 --- [ocp-schedules-17,ee4caec197cc8622,9254c763523138b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:11:57.844 INFO 2458 --- [ocp-schedules-17,ee4caec197cc8622,9254c763523138b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:11:58.513 INFO 2458 --- [ocp-async-8,0d382fc955d4e943,20812ac5cce4cadc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:11:58.522 WARN 2458 --- [ocp-async-6,587d58e2b0bf5ad6,f150d0964094d04f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:11:58.522 INFO 2458 --- [ocp-async-4,f907114729410102,af0162dd48a579b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:11:59.240 INFO 2458 --- [ocp-schedules-14,5c6b738baa99a346,91510e3643d431d9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:11:59.707 INFO 2458 --- [prometheus-minute-schedule-1,d3ca2b9a02165076,8b4338040e8dfcd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534720, interval=60 2025-10-27 11:11:59.715 INFO 2458 --- [prometheus-minute-schedule-1,d3ca2b9a02165076,8b4338040e8dfcd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:00.053 INFO 2458 --- [ocp-schedules-10,4c57b6b32300ba46,58a7f0c5de738212] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:12:00.060 INFO 2458 --- [ocp-schedules-10,4c57b6b32300ba46,58a7f0c5de738212] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:00.075 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:00.076 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:00.163 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:12:00.163 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.167 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:12:00.180 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.180 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.209 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.209 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.227 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.232 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:12:00.239 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.240 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.242 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.242 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.269 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.269 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.282 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.282 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.282 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.283 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.295 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.295 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.311 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.311 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.312 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.312 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:12:00.314 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.314 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.314 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.315 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.330 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.331 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.332 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.333 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.334 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.335 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.335 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.337 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.337 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.338 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.339 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.340 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.341 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:12:00.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.343 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.344 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.344 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.347 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:12:00.372 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.372 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.372 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.373 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:12:00.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.375 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.376 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:12:00.410 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.410 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.410 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.411 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:12:00.437 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:00.437 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.438 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:12:00.446 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.446 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.446 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.447 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.449 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.450 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:12:00.453 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:12:00.453 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:12:00.478 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.478 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.479 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.479 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:12:00.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.507 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.507 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:12:00.522 WARN 2458 --- [ocp-async-9,a8932b3981b5b25a,c5d7edc9d80ae425] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:00.522 INFO 2458 --- [ocp-async-10,8f5cb37c99887a74,f58aeaa03b40a519] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:00.562 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.562 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.563 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.564 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.564 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.564 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.564 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.565 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.565 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.565 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.566 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.566 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.567 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.567 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.567 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.568 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.569 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.569 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.569 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.570 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.570 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.571 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.571 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.571 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.572 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.572 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.573 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.574 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.574 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.574 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.575 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.575 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.575 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.575 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.576 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:12:00.576 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.576 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.576 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.577 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:12:00.578 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.578 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.578 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.579 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:12:00.579 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:12:00.582 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:12:00.669 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:00.670 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.670 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.670 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:00.670 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:00.670 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:00.671 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:12:00.698 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:12:00.702 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:12:00.702 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:12:00.703 INFO 2458 --- [ocp-schedules-10,4c57b6b32300ba46,58a7f0c5de738212] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:12:01.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:01.716457+08:00, lastProcessTime=2025-10-27T11:11:51.713752+08:00 2025-10-27 11:12:01.935 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:01.716457+08:00 2025-10-27 11:12:02.519 INFO 2458 --- [ocp-schedules-10,889ebba980ed5e78,0b5717053c57506c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:12:02.523 WARN 2458 --- [ocp-async-2,394adb1028bb1403,802b934dc610a25e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:02.523 INFO 2458 --- [ocp-async-5,83fd4e4fb4fe87ae,97d1a35fe31df433] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:02.526 INFO 2458 --- [ocp-schedules-10,889ebba980ed5e78,0b5717053c57506c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:02.569 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:02.569 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:12:02.590 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:02.590 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:02.590 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:02.590 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:02.590 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:02.591 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:12:02.608 INFO 2458 --- [ocp-schedules-10,889ebba980ed5e78,0b5717053c57506c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:12:02.707 INFO 2458 --- [prometheus-minute-schedule-1,5e0fdbbef9d66c9b,f41493e1b7bf32e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:02.717 INFO 2458 --- [prometheus-minute-schedule-1,5e0fdbbef9d66c9b,f41493e1b7bf32e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:03.076 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:03.077 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:03.914 INFO 2458 --- [ocp-schedules-19,dc5593868d90e641,2654644b14cfd097] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:03.917 INFO 2458 --- [ocp-schedules-19,dc5593868d90e641,2654644b14cfd097] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:04.523 INFO 2458 --- [ocp-async-7,d382c5ff1a57f085,3f8b090c25f8366f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:04.523 WARN 2458 --- [ocp-async-8,b2cc63f6e1f6e1ce,8ce27a0bfa70b178] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:05.707 INFO 2458 --- [prometheus-minute-schedule-1,6b6fde5f627c6796,4cd4a6f511e6fdde] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:05.715 INFO 2458 --- [prometheus-minute-schedule-1,6b6fde5f627c6796,4cd4a6f511e6fdde] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:06.024 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:06.025 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:12:06.027 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:06.027 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:12:06.029 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:12:06.057 INFO 2458 --- [ocp-schedules-16,cec41330d93ee9d9,141e3ae235c852bf] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:06.077 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:06.078 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:06.085 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:12:06.126 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:12:06.152 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:12:06.152 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:12:06.152 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:12:06.152 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-27T11:09:30, target=2025-10-27T11:07:30, granule=120000ms, delay=120000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:07:30, target=2025-10-27T11:02:30, granule=600000ms, delay=300000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:12:06.155 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=126ms, period=120000ms 2025-10-27 11:12:06.523 INFO 2458 --- [ocp-async-7,cbab67f545bf0fb7,a01a3a2394d1c97c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:06.523 WARN 2458 --- [ocp-async-4,71211700e597fd27,9b375c3152c30e2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:06.880 INFO 2458 --- [ocp-schedules-26,c27ee00463bcc6d7,5359c8986942df25] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:12:06.888 INFO 2458 --- [ocp-schedules-26,c27ee00463bcc6d7,5359c8986942df25] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:06.888 INFO 2458 --- [ocp-schedules-26,c27ee00463bcc6d7,5359c8986942df25] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:07.064 INFO 2458 --- [ocp-schedules-8,a485d7595f394ada,9e030839a90e0c6e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:07.096 WARN 2458 --- [scanner-112146951-1,a485d7595f394ada,ae9275c2a83206b8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:08.513 INFO 2458 --- [ocp-async-7,b7d1f5ed694bf4cf,8ee09a255e5d8a59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:08.523 INFO 2458 --- [ocp-async-10,f7d3d39ed74d205a,aea406e9fe86275a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:08.523 WARN 2458 --- [ocp-async-6,f9c1630d3f4cc09a,493286f0564c9696] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:08.707 INFO 2458 --- [prometheus-minute-schedule-1,82e9d11597eaed98,429487dd83feab19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:08.715 INFO 2458 --- [prometheus-minute-schedule-1,82e9d11597eaed98,429487dd83feab19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:09.079 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:09.080 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:10.014 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:12:10.014 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:12:10.052 INFO 2458 --- [Collect obproxy1,e6cc1c098a5a67ad,dd5a2289c39edff7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:12:10.052 INFO 2458 --- [Check obproxy1,f9237be20d33b609,bcf531d49300a1ac] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:12:10.052 INFO 2458 --- [Check obproxy1,f9237be20d33b609,bcf531d49300a1ac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:12:10.052 INFO 2458 --- [Collect obproxy1,e6cc1c098a5a67ad,dd5a2289c39edff7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:12:10.082 INFO 2458 --- [Check obproxy1,f9237be20d33b609,bcf531d49300a1ac] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:12:10.085 INFO 2458 --- [Check obproxy1,f9237be20d33b609,bcf531d49300a1ac] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:12:10.085 INFO 2458 --- [Check obproxy1,f9237be20d33b609,bcf531d49300a1ac] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:12:10.086 INFO 2458 --- [Collect obproxy1,e6cc1c098a5a67ad,dd5a2289c39edff7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:12:10.089 INFO 2458 --- [Collect obproxy1,e6cc1c098a5a67ad,dd5a2289c39edff7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:12:10.092 INFO 2458 --- [Collect obproxy1,e6cc1c098a5a67ad,dd5a2289c39edff7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:12:10.184 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:12:10.523 WARN 2458 --- [ocp-async-1,f79a069b7da85848,3bb710940dc1d12f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:10.523 INFO 2458 --- [ocp-async-8,f3b489be4089f46f,0ca494ddc9a53eb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:11.707 INFO 2458 --- [prometheus-minute-schedule-1,08950a703edd77c0,5d91dcca51bbbe0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:11.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:11.715202+08:00, lastProcessTime=2025-10-27T11:12:01.716457+08:00 2025-10-27 11:12:11.716 INFO 2458 --- [prometheus-minute-schedule-1,08950a703edd77c0,5d91dcca51bbbe0b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:11.959 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:11.715202+08:00 2025-10-27 11:12:12.080 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:12.081 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:12.523 WARN 2458 --- [ocp-async-8,3ea8a223c87f0362,3691559ef6e5f3b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:12.523 INFO 2458 --- [ocp-async-8,bbaa307e8c3e99a5,a4bc91aa078738cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:13.749 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534660, nodeCount=817, affectRows=817 2025-10-27 11:12:13.925 INFO 2458 --- [ocp-schedules-24,416334c12e2add1a,fc1b94398cad7fcc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:13.930 INFO 2458 --- [ocp-schedules-24,416334c12e2add1a,fc1b94398cad7fcc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:14.524 INFO 2458 --- [ocp-async-5,35bb72478b15e5c4,1f4df4af9d08f79f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:14.524 WARN 2458 --- [ocp-async-7,edaa94d9c662a894,ee3f9f10c0ad56b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:14.657 INFO 2458 --- [ocp-schedules-2,a656d15e6e31feb1,42a07a3fbb0148e2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:12:14.657+08:00, clusters=[1, 2] 2025-10-27 11:12:14.666 INFO 2458 --- [ocp-schedules-2,a656d15e6e31feb1,42a07a3fbb0148e2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:12:14.667 INFO 2458 --- [ocp-schedules-2,a656d15e6e31feb1,42a07a3fbb0148e2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:12:14.707 INFO 2458 --- [prometheus-minute-schedule-1,7606e9d34f3bd1e3,d2c0e6788f41d758] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:14.715 INFO 2458 --- [prometheus-minute-schedule-1,7606e9d34f3bd1e3,d2c0e6788f41d758] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:14.927 INFO 2458 --- [ocp-schedules-10,eeae67d9fe341cbb,86a553fd0158a34c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:12:14.927 INFO 2458 --- [ocp-schedules-10,eeae67d9fe341cbb,86a553fd0158a34c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:12:15.000 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:12:15.000 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:12:15.001 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:12:15.001 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:12:15.001 INFO 2458 --- [Collect all tenants major compactions6,eeae67d9fe341cbb,160d5ce9805d7c96] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:12:15.002 INFO 2458 --- [Collect all tenants major compactions5,eeae67d9fe341cbb,f6bde44635a8b1d8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:12:15.043 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587936) Duplicate entry '835014129065861120-1761534600' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587936) Duplicate entry '835014129065861120-1761534600' for key 'PRIMARY' 2025-10-27 11:12:15.044 INFO 2458 --- [Collect all tenants major compactions5,eeae67d9fe341cbb,f6bde44635a8b1d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.046 INFO 2458 --- [Collect all tenants major compactions5,eeae67d9fe341cbb,f6bde44635a8b1d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:12:15.053 INFO 2458 --- [Collect all tenants major compactions5,eeae67d9fe341cbb,f6bde44635a8b1d8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:12:15.060 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=1000, affectRows=1000 2025-10-27 11:12:15.063 INFO 2458 --- [Collect all tenants major compactions6,eeae67d9fe341cbb,160d5ce9805d7c96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.064 INFO 2458 --- [Collect all tenants major compactions6,eeae67d9fe341cbb,160d5ce9805d7c96] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:12:15.065 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=994, validCount=992 2025-10-27 11:12:15.074 INFO 2458 --- [Collect all tenants major compactions6,eeae67d9fe341cbb,160d5ce9805d7c96] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:12:15.081 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:15.082 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:15.089 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=1000, affectRows=1000 2025-10-27 11:12:15.094 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=996, validCount=994 2025-10-27 11:12:15.096 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=1000, affectRows=1000 2025-10-27 11:12:15.101 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=999, validCount=995 2025-10-27 11:12:15.102 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.102 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.102 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.102 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.103 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.104 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:12:15.104 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:12:15.105 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:12:15.108 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.108 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.108 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:15.108 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:15.108 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:15.110 INFO 2458 --- [Collect all tenants major compactions4,eeae67d9fe341cbb,ac0ba32938a19c4e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:12:15.110 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:12:15.110 INFO 2458 --- [Collect all tenants major compactions1,eeae67d9fe341cbb,348f3f373d578c9c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:12:15.111 INFO 2458 --- [Collect all tenants major compactions3,eeae67d9fe341cbb,1d67c20953c309ea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:12:15.116 INFO 2458 --- [Collect all tenants major compactions2,eeae67d9fe341cbb,c3c5d3231ea57f38] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:12:15.126 INFO 2458 --- [ocp-schedules-10,eeae67d9fe341cbb,86a553fd0158a34c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:12:15.281 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276} 2025-10-27 11:12:15.284 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=76, affectRows=76 2025-10-27 11:12:15.284 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=75, validCount=75 2025-10-27 11:12:15.286 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=1000, affectRows=1000 2025-10-27 11:12:15.291 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=1000, validCount=995 2025-10-27 11:12:15.292 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameI 2025-10-27 11:12:15.292 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.296 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.296 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:12:15.296 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.299 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.300 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.300 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:12:15.300 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.303 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.304 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.304 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:12:15.304 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.307 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.330 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 27 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.400 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:12:15.400 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.522 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448335521000, endTimeUs=1761534615276000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:12:15.521Z, endTime=2025-10-27T03:10:15.276Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:12:15.541 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:12:15.541 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:12:15.566 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:12:15.566 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534615276000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:12:15.630 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:12:15.630 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:12:15.666 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:12:15.666 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534615276000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:12:15.674 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:12:15.674 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:12:15.521, end=2025-10-27T11:10:15.276}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:12:15.674 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.676 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.676 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:12:15.676 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.679 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:12:15.682 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:08:14.38, end=2025-10-27T11:10:15.276},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:12:15.682 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:12:15.682 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:12:15.689 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:12:16.064 INFO 2458 --- [ocp-schedules-28,a33d964e4db3d286,019a299cccf6d4df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:16.524 INFO 2458 --- [ocp-async-1,2e3fcf2744971f9f,75860a7af7e09e9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:16.524 WARN 2458 --- [ocp-async-9,2e95fa9a6be8f42d,e29e2381ee37f447] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:16.630 INFO 2458 --- [ocp-schedules-10,a0a58fe0cd63b986,ee7b4b6b38dadb34] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:12:16.630 INFO 2458 --- [ocp-schedules-10,a0a58fe0cd63b986,ee7b4b6b38dadb34] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:12:16.638 INFO 2458 --- [ocp-schedules-10,a0a58fe0cd63b986,ee7b4b6b38dadb34] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:12:16.638 INFO 2458 --- [ocp-schedules-10,a0a58fe0cd63b986,ee7b4b6b38dadb34] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:12:17.217 INFO 2458 --- [ocp-schedules-5,840cf304fc86ae1a,f1ae21ba503cee16] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:17.262 WARN 2458 --- [scanner-112148718-1,840cf304fc86ae1a,b8b1350095144a6c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:17.397 INFO 2458 --- [alarm-detect-16,840cf304fc86ae1a,215acba2645f28b0] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534705 ~ 1761534727 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534720, value=75.474)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534720, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534720=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:12:17.707 INFO 2458 --- [prometheus-minute-schedule-1,8c3a07cefa8192da,0671ed5648e23bfc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:17.714 INFO 2458 --- [prometheus-minute-schedule-1,8c3a07cefa8192da,0671ed5648e23bfc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:18.082 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:18.083 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:18.514 INFO 2458 --- [ocp-async-4,16e7728eda40144e,61c74ba0c698dc8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:18.524 WARN 2458 --- [ocp-async-2,9d87c2bfdb909c58,e19ff3db079b4864] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:18.524 INFO 2458 --- [ocp-async-2,e04352d4a8e91e07,337441c103aac9f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:19.457 INFO 2458 --- [ocp-schedules-10,c94902b972358584,08a4f6e950afe55d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:12:19.970 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534660, nodeCount=1000, affectRows=1000 2025-10-27 11:12:19.974 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534660, nodeCount=996, validCount=983 2025-10-27 11:12:20.524 WARN 2458 --- [ocp-async-7,0b25fa47ed924720,d1761ba9bc43d263] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:20.524 INFO 2458 --- [ocp-async-6,148346c631f255f8,5e0739e266e6de23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:20.707 INFO 2458 --- [prometheus-minute-schedule-1,ac8c14808a4fcb48,a386400c3b34c555] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:20.715 INFO 2458 --- [prometheus-minute-schedule-1,ac8c14808a4fcb48,a386400c3b34c555] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:21.083 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:21.084 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:21.411 INFO 2458 --- [ocp-schedules-15,8f1ac6f706d78954,3aba8aef9441540e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:21.412 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:12:21.412 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:12:21.412 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:12:21.413 INFO 2458 --- [ocp-schedules-25,540cce0931e079df,148e55c9b07ee45f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:21.413 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:12:21.414 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:12:21.428 INFO 2458 --- [ocp-schedules-29,bf0a1ce042106813,dc1a4797c05190e7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:21.429 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:21.430 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:21.430 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:21.431 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:21.431 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:21.431 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:21.431 INFO 2458 --- [host-check-clockdiff1,bf0a1ce042106813,929e5ce99770525a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:12:21.433 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:21.434 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:21.434 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:21.434 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:21.435 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:21.435 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:21.436 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:21.436 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:21.436 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:21.441 INFO 2458 --- [host-check-clockdiff2,bf0a1ce042106813,2613ef3eddcc2ec3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:12:21.441 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:21.442 INFO 2458 --- [ocp-schedules-17,7b327d7dede5d3f9,1f192923fb0fdda6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:12:21.442 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:21.442 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:21.442 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:21.459 INFO 2458 --- [host-check-clockdiff3,bf0a1ce042106813,340c88ec382c4bb6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:12:21.459 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:21.460 INFO 2458 --- [host-check-clockdiff4,bf0a1ce042106813,7fdffe0ec4fe8cde] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:12:21.463 INFO 2458 --- [host-check-clockdiff5,bf0a1ce042106813,e80181f1f3152840] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:12:21.484 INFO 2458 --- [host-check-clockdiff1,bf0a1ce042106813,929e5ce99770525a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534741 0 0 , error= 2025-10-27 11:12:21.494 INFO 2458 --- [host-check-clockdiff2,bf0a1ce042106813,2613ef3eddcc2ec3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534741 -1 -1 , error= 2025-10-27 11:12:21.498 INFO 2458 --- [host-check-clockdiff5,bf0a1ce042106813,e80181f1f3152840] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534741 0 0 , error= 2025-10-27 11:12:21.498 INFO 2458 --- [host-check-clockdiff4,bf0a1ce042106813,7fdffe0ec4fe8cde] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534741 -1 -1 , error= 2025-10-27 11:12:21.526 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:12:21.531 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:12:21.531 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:12:21.530044626+08:00 2025-10-27 11:12:21.532 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:12:21.531+08:00 2025-10-27 11:12:21.533 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:12:21.532202381+08:00 2025-10-27 11:12:21.533 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:12:21.533+08:00 2025-10-27 11:12:21.534 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:12:21.535 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:12:21.536 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:12:21.536 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:12:21.536 INFO 2458 --- [host-check-agent-status4,540cce0931e079df,b5efa89ec8f27ad0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:12:21.537 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:12:21.539 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:12:21.538155778+08:00 2025-10-27 11:12:21.539 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:12:21.540 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:12:21.539+08:00 2025-10-27 11:12:21.540 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:12:21.540 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:12:21.540 INFO 2458 --- [host-check-agent-status3,540cce0931e079df,e51271c84144e5e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:12:21.541 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:12:21.541 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:12:21.542 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:12:21.542 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:12:21.542 INFO 2458 --- [host-check-agent-status2,540cce0931e079df,4ba361b8a3419149] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:12:21.542 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:12:21.531+08:00 and status AVAILABLE 2025-10-27 11:12:21.543 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:12:21.543502468+08:00 2025-10-27 11:12:21.544 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:12:21.533+08:00 and status AVAILABLE 2025-10-27 11:12:21.544 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:12:21.543+08:00 2025-10-27 11:12:21.545 INFO 2458 --- [ocp-schedules-12,4d6cb436e046500b,ba09fdaad2e379f9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:12:21.545 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:12:21.545 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:12:21.545 INFO 2458 --- [host-check-agent-status1,540cce0931e079df,d4877700990abbc3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:12:21.547 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:12:21.547 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:12:21.547 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:12:21.547602950+08:00 2025-10-27 11:12:21.548 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:12:21.547+08:00 2025-10-27 11:12:21.549 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:12:21.549 INFO 2458 --- [ocp-schedules-26,59401174166719ef,e31c2ff4f411bd91] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:12:21.549 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:12:21.539+08:00 and status AVAILABLE 2025-10-27 11:12:21.549 INFO 2458 --- [ocp-schedules-12,4d6cb436e046500b,ba09fdaad2e379f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:21.550 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:12:21.552 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:12:21.543+08:00 and status AVAILABLE 2025-10-27 11:12:21.552 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:12:21.552 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:12:21.552 INFO 2458 --- [host-check-agent-status5,540cce0931e079df,18e64b6805f063d7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:12:21.553 INFO 2458 --- [ocp-schedules-26,59401174166719ef,e31c2ff4f411bd91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:21.554 INFO 2458 --- [ocp-schedules-11,7c7cee5604747ddd,31a88c3d6e6516d8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:12:21.554 INFO 2458 --- [ocp-schedules-26,59401174166719ef,e31c2ff4f411bd91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:12:21.555 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:12:21.555 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:12:21.547+08:00 and status AVAILABLE 2025-10-27 11:12:21.557 INFO 2458 --- [ocp-schedules-11,7c7cee5604747ddd,31a88c3d6e6516d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:21.558 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:12:21.560 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:12:21.568 INFO 2458 --- [ocp-schedules-13,23fafe6118e7fcbd,e41e6a9c38967a85] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:12:21.571 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:21.572 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:21.573 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:21.575 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:21.579 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:21.581 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:21.582 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:21.584 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:21.588 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:21.592 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:21.606 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.607 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:21.608 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:12:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.640 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:21.645 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:12:21.645 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.646 INFO 2458 --- [ocp-schedules-12,4d6cb436e046500b,ba09fdaad2e379f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:12:21.646 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:21.661 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:21.662 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:21.666 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:21.668 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:21.677 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:12:21.682 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:12:21.682 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:21.682 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:12:21.691 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:12:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.702 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:21.702 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.702 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:21.702 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.703 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:12:21.711 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:12:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:21.715124+08:00, lastProcessTime=2025-10-27T11:12:11.715202+08:00 2025-10-27 11:12:21.719 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:12:21.724 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:12:21.726 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:12:21.731 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:12:21.735 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:21.735 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2025-10-27 11:12:21.735 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2025-10-27 11:12:21.735 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2025-10-27 11:12:21.735 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2025-10-27 11:12:21.735 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2025-10-27 11:12:21.754 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : binlogCluster BinlogCluster(id=4, name=binlogtest2, version=4.0.1-100000162024122511, status=RUNNING, metadbInfo=BinlogClusterMetadbInfo(tenantId=2, username=root, database=binlog_database, tenantName=ocp_meta, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187, clusterType=PRIMARY, obVersion=4.3.5.4, address=10.40.65.146, port=2881, fullUserName=root@ocp_meta), resourceStats=null, servers=[BinlogServer(id=4, ip=10.40.65.151, port=2983, binlogClusterId=4, binlogClusterName=binlogtest2, hostId=4, hostName=tjtx40-65-151.zhuanos.com, hostKind=DEDICATED_PHYSICAL_MACHINE, hostTypeName=test, idcName=default_idc, regionName=deault_region, status=RUNNING, resourceStats=null)], instances=null, tags=null, createTime=2025-10-24T17:01:22+08:00, updateTime=2025-10-24T17:04:41+08:00, installPath=/home/admin/binlogservice, binlogFilePath=/home/admin/binlogservice/run, address=10.40.65.151, port=2983) 2025-10-27 11:12:21.755 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:12:21.762 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:21.763 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:21.770 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:21.775 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:21.776 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:21.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.777 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:21.778 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.780 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:12:21.781 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:12:21.783 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:12:21.784 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:21.788 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:12:21.796 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:21.796 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:12:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:21.802 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:21.803 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.803 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:12:21.835 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:21.835 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:21.836 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:12:21.843 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:12:21.843 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:12:21.854 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:12:21.862 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:12:21.862 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:12:21.862 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.877 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:12:21.888 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2025-10-27 11:12:21.889 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:12:21.895 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:21.899 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2025-10-27 11:12:21.899 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:21.904 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:21.907 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:21.910 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:21.912 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:21.915 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:12:21.915 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:12:21.915 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:12:21.917 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:12:21.917 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:12:21.917 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:12:21.920 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:12:21.920 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2025-10-27 11:12:21.921 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2025-10-27 11:12:21.923 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:12:21.924 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:21.932 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2025-10-27 11:12:21.935 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:12:21.935 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2 to set sql collect property inactive 2025-10-27 11:12:21.935 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:21.937 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:21.941 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:21.943 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:21.943 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:21.950 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:21.958 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:12:21.966 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2025-10-27 11:12:21.967 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:21.968 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:21.969 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:21.970 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:12:21.971 INFO 2458 --- [host-check-host-status5,8f1ac6f706d78954,ee67132842ff4c29] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:12:21.971 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:12:21.977 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 3 to set sql collect property inactive 2025-10-27 11:12:21.980 INFO 2458 --- [host-check-host-status3,8f1ac6f706d78954,504097410ab8b59c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:12:21.980 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:21.987 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:21.987 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:12:21.999 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:22.000 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:22.005 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:22.007 INFO 2458 --- [host-check-host-status4,8f1ac6f706d78954,953be15bb5f28eef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:12:22.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:22.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.018 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:22.020 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.020 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.021 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.024 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.binlog.cluster.id=4), (monagent.binlog.cluster.name=binlogtest2), (monagent.pipeline.logproxy.status=active), (binlog.multi.instance.config={"instanceConfigs":[],"instanceType":"binlog"}), (binlog.install.path=/home/admin/binlogservice), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_hardware_error","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["Hardware Error"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (mo 2025-10-27 11:12:22.024 INFO 2458 --- [host-check-host-status2,8f1ac6f706d78954,8b3b52b29d84c812] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:12:22.024 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:12:22.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:22.034 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:12:22.049 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.146), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.146:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:12:22.049 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.168), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.168:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:12:22.054 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2025-10-27 11:12:22.059 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510188), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.66.107), (ob.run.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.66.107:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase2), (ob.logtailer.enabled=true), (ob.l 2025-10-27 11:12:22.091 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2025-10-27 11:12:22.092 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2025-10-27 11:12:22.094 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2025-10-27 11:12:22.106 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:12:22.110 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.67.22), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.67.22:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtailer. 2025-10-27 11:12:22.112 INFO 2458 --- [host-check-host-status1,8f1ac6f706d78954,6b0ed2d952a37090] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:12:22.114 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.114 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:22.128 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2025-10-27 11:12:22.128 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:21.715124+08:00 2025-10-27 11:12:22.130 INFO 2458 --- [ocp-schedules-30,d8e5c437e07006d1,554e2dad19f42940] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-10-27 11:12:22.148 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:12:22.149 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.149 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:12:22.149 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.150 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:22.151 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.152 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:12:22.167 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:12:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:22.190 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:22.190 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.191 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:12:22.213 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:12:22.213 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:12:22.230 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.230 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:22.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:22.300 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.344 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.345 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:22.349 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:12:22.360 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:12:22.370 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:22.411 INFO 2458 --- [host-check-clockdiff3,bf0a1ce042106813,340c88ec382c4bb6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534742 0 0 , error= 2025-10-27 11:12:22.437 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:12:22.489 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:12:22.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.489 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.490 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:12:22.492 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.492 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.492 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.493 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:12:22.524 WARN 2458 --- [ocp-async-1,c7d3a6296d350fbb,2c2154a6593ee45a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:22.524 INFO 2458 --- [ocp-async-10,ade3fa343f9ab498,cdb3552efeecaf01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:22.526 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:12:22.574 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:22.574 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.574 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.575 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:22.575 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:22.575 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:22.575 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:12:22.629 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:12:22.629 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:12:22.629 INFO 2458 --- [ocp-schedules-11,7c7cee5604747ddd,31a88c3d6e6516d8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:12:23.707 INFO 2458 --- [prometheus-minute-schedule-1,16ac019e7a0a6d08,ef13a265724d523e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:23.716 INFO 2458 --- [prometheus-minute-schedule-1,16ac019e7a0a6d08,ef13a265724d523e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:23.937 INFO 2458 --- [ocp-schedules-28,7ff92758a6f07e43,38017ebce5cb57a7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:23.941 INFO 2458 --- [ocp-schedules-28,7ff92758a6f07e43,38017ebce5cb57a7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:24.085 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:24.086 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:24.524 WARN 2458 --- [ocp-async-3,0920f315910bfc63,ac33ff79492082bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:24.524 INFO 2458 --- [ocp-async-3,980291310d662db9,df2d60748a80be01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:26.074 INFO 2458 --- [ocp-schedules-7,a25171b9c28afdcc,4650b28f43d6a392] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:26.524 INFO 2458 --- [ocp-async-2,4c4486c5ae24a351,85c26f4381c69bb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:26.524 WARN 2458 --- [ocp-async-7,d9ac907efdb63c50,5772554d12f5b80d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:26.707 INFO 2458 --- [prometheus-minute-schedule-1,d5494c947d5670fd,1dc0a76e1efb8a40] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:26.715 INFO 2458 --- [prometheus-minute-schedule-1,d5494c947d5670fd,1dc0a76e1efb8a40] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:26.892 INFO 2458 --- [ocp-schedules-3,64b40e3729d1a0ed,dd27c2b9f26597e2] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:12:26.901 INFO 2458 --- [ocp-schedules-3,64b40e3729d1a0ed,dd27c2b9f26597e2] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:26.901 INFO 2458 --- [ocp-schedules-3,64b40e3729d1a0ed,dd27c2b9f26597e2] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:27.086 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:27.087 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:27.177 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b9fc04fc63e114cd,423f5a0a6c32322c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b9fc04fc63e114cd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:12:27.178 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b9fc04fc63e114cd,423f5a0a6c32322c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b9fc04fc63e114cd, duration=1 ms] 2025-10-27 11:12:27.405 INFO 2458 --- [ocp-schedules-25,08bd5e27f6fea7e9,b30b74ef2693cc75] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:27.442 WARN 2458 --- [scanner-112148930-1,08bd5e27f6fea7e9,7b18761db90cb033] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:27.682 INFO 2458 --- [alarm-detect-8,08bd5e27f6fea7e9,b67258437f99e337] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534720 ~ 1761534737 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534735, value=88.719)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534735, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534735=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:12:28.514 INFO 2458 --- [ocp-async-6,aa8d157eca31943c,5dd4ac119218666c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:28.524 INFO 2458 --- [ocp-async-1,c47492172bb615fd,a2449e9a11c11f2a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:28.524 WARN 2458 --- [ocp-async-10,59c0c6d2aefddbe6,1371486bb413f64a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:28.874 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:12:28.874 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:12:28.882 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:12:28.882 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:12:28.889 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:12:28.889 INFO 2458 --- [ocp-schedules-7,5ecca8d0c148de1a,87db75ba059a2cf2] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:12:29.350 INFO 2458 --- [ocp-schedules-10,148421ebfe0ee182,860c3dd2db52f824] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:12:29.356 INFO 2458 --- [ocp-schedules-10,148421ebfe0ee182,860c3dd2db52f824] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:12:29.356 INFO 2458 --- [ocp-schedules-10,148421ebfe0ee182,860c3dd2db52f824] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:12:29.707 INFO 2458 --- [prometheus-minute-schedule-1,d2b5c8cd24288d7a,a668dd12504bc284] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:29.715 INFO 2458 --- [prometheus-minute-schedule-1,d2b5c8cd24288d7a,a668dd12504bc284] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:30.087 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:30.088 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:30.525 INFO 2458 --- [ocp-async-8,ab346de96afae7d7,72560ce778d2a255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:30.525 WARN 2458 --- [ocp-async-3,f1b99f5821f3397e,929ccc2e502d3c1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:31.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:31.718165+08:00, lastProcessTime=2025-10-27T11:12:21.715124+08:00 2025-10-27 11:12:31.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:31.718165+08:00 2025-10-27 11:12:32.525 WARN 2458 --- [ocp-async-5,855f0b17f0468581,1802008ad9dc2ab6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:32.525 INFO 2458 --- [ocp-async-5,daa7a0ba6dd9992e,934f983ae3ef15f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:32.707 INFO 2458 --- [prometheus-minute-schedule-1,685fba14b9eb47f9,1b8c01e1d01a2f69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:32.716 INFO 2458 --- [prometheus-minute-schedule-1,685fba14b9eb47f9,1b8c01e1d01a2f69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:33.088 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:33.089 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:33.778 INFO 2458 --- [ocp-schedules-5,97c45a3cc2c8e58c,3ae3766d3ab7a877] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:12:33.778 INFO 2458 --- [ocp-schedules-5,97c45a3cc2c8e58c,3ae3766d3ab7a877] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:12:33.948 INFO 2458 --- [ocp-schedules-9,842269dda9887cd1,7c7b2cba90b889fc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:33.952 INFO 2458 --- [ocp-schedules-9,842269dda9887cd1,7c7b2cba90b889fc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:34.053 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:12:34.053 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:12:34.073 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:12:34.073 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:12:34.073 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:12:34.090 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:12:34.090 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:12:34.090 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:12:34.101 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:12:34.101 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:12:34.101 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:12:34.112 INFO 2458 --- [ocp-schedules-30,c3ba327236f5fc43,ba3e0926d072142c] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:12:34.525 WARN 2458 --- [ocp-async-9,6dacd3aee85d3139,74f61867cf991886] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:34.525 INFO 2458 --- [ocp-async-6,51132a606b7697b5,c1ff8db0b8ae4808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:35.707 INFO 2458 --- [prometheus-minute-schedule-1,3ec1ea2a36b63bac,58fb44e0e7b0779d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:35.716 INFO 2458 --- [prometheus-minute-schedule-1,3ec1ea2a36b63bac,58fb44e0e7b0779d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:36.008 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:12:36.008 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:12:36.082 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:36.082 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:36.082 INFO 2458 --- [ocp-schedules-23,4b8d7799aab49be8,f017ca84cad16299] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:36.089 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:36.090 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:36.120 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:36.120 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:36.120 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:36.121 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:12:36.133 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:36.133 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:36.135 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:36.135 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:12:36.199 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:36.200 INFO 2458 --- [ocp-schedules-12,b10be53e2638bbf5,2f13ba58709d5a37] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:12:36.525 WARN 2458 --- [ocp-async-3,354a11f36b588e24,854886ebaa090cfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:36.525 INFO 2458 --- [ocp-async-4,ac7e5a5809c9b4be,8812a1fd1d24b363] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:37.672 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:37.672 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:12:37.672 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:12:37.672 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:12:37.672 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:12:37.672 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:12:37.688 INFO 2458 --- [ocp-schedules-4,fcafc07e4690fdd1,8dde536a995879b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:37.693 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:37.697 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:37.699 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:37.702 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:37.705 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:37.706 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:37.707 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:37.708 INFO 2458 --- [prometheus-cache-monitor1,8c35fb28b33bd1d0,f3b878cfd59fcf3c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20307,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5077,"seriesIdKeySize":8770,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:12:37.708 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:37.708 INFO 2458 --- [prometheus-cache-monitor1,8c35fb28b33bd1d0,f3b878cfd59fcf3c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:12:37.709 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:37.710 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:37.793 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:12:37.794 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:12:37.802 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:12:37.803 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:12:37.805 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:12:37.807 INFO 2458 --- [ocp-schedules-12,86dcddfa89b59e23,d814744c33be56b9] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:12:37.814 WARN 2458 --- [scanner-112149179-1,fcafc07e4690fdd1,df7e504a6dd563e5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:38.514 INFO 2458 --- [ocp-async-4,e9e90215ebb49703,a5eefd455515a952] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:38.525 WARN 2458 --- [ocp-async-2,8f42079af057ea7c,835153429915bce1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:38.525 INFO 2458 --- [ocp-async-5,2f73025318653d81,0261e8bf165ee5a7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:38.626 INFO 2458 --- [ocp-schedules-8,b07a914b0c991f51,a7a660e06143a741] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:12:38.633 INFO 2458 --- [ocp-schedules-8,b07a914b0c991f51,a7a660e06143a741] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:38.633 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:12:38.633 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:12:38.634 INFO 2458 --- [ocp-schedules-8,b07a914b0c991f51,a7a660e06143a741] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:12:38.707 INFO 2458 --- [prometheus-minute-schedule-1,22b7f67bd7e28514,39590137fc0de8b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:38.715 INFO 2458 --- [prometheus-minute-schedule-1,22b7f67bd7e28514,39590137fc0de8b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:39.090 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:39.091 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:40.013 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:12:40.013 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:12:40.048 INFO 2458 --- [Check obproxy1,5a8626fed3b2d044,1f591c28c653aab6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:12:40.048 INFO 2458 --- [Check obproxy1,5a8626fed3b2d044,1f591c28c653aab6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:12:40.048 INFO 2458 --- [Collect obproxy1,406c5fc4daaf00d8,55b18c10cdc34541] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:12:40.048 INFO 2458 --- [Collect obproxy1,406c5fc4daaf00d8,55b18c10cdc34541] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:12:40.078 INFO 2458 --- [Check obproxy1,5a8626fed3b2d044,1f591c28c653aab6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:12:40.079 INFO 2458 --- [Collect obproxy1,406c5fc4daaf00d8,55b18c10cdc34541] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:12:40.080 INFO 2458 --- [Check obproxy1,5a8626fed3b2d044,1f591c28c653aab6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:12:40.080 INFO 2458 --- [Check obproxy1,5a8626fed3b2d044,1f591c28c653aab6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:12:40.082 INFO 2458 --- [Collect obproxy1,406c5fc4daaf00d8,55b18c10cdc34541] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:12:40.085 INFO 2458 --- [Collect obproxy1,406c5fc4daaf00d8,55b18c10cdc34541] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:12:40.525 WARN 2458 --- [ocp-async-9,af60070074c60001,07c6d2f8f74c6bcd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:40.525 INFO 2458 --- [ocp-async-10,22a7fb580c1441e5,1354225f1f9f45c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:40.842 INFO 2458 --- [ocp-schedules-15,668a1b1ccd4459f6,18f568c045f16ce8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:12:40.843 INFO 2458 --- [ocp-schedules-15,668a1b1ccd4459f6,18f568c045f16ce8] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:12:40.866 INFO 2458 --- [ocp-schedules-15,668a1b1ccd4459f6,18f568c045f16ce8] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] skip sync tenant binlog service=9, status=CREATING 2025-10-27 11:12:41.707 INFO 2458 --- [prometheus-minute-schedule-1,b7a489e3ad89c3f3,29f3f0f3e497a3d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:41.714 INFO 2458 --- [prometheus-minute-schedule-1,b7a489e3ad89c3f3,29f3f0f3e497a3d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:41.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:41.716521+08:00, lastProcessTime=2025-10-27T11:12:31.718165+08:00 2025-10-27 11:12:41.920 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:41.716521+08:00 2025-10-27 11:12:42.092 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:42.093 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:42.525 INFO 2458 --- [ocp-async-3,e899094e1df84624,5a810ff85ce6b5cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:42.525 WARN 2458 --- [ocp-async-1,ec7c6ee4813720de,24b136dc59f06426] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:43.960 INFO 2458 --- [ocp-schedules-4,eb14d01e71ea0eed,5859fd014900314a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:43.963 INFO 2458 --- [ocp-schedules-4,eb14d01e71ea0eed,5859fd014900314a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:44.036 INFO 2458 --- [ocp-schedules-10,0f64b60469756b26,1a59e66495ac3bdf] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:12:44.138 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ab6b4757201d1008,5b6dfbe6d81e411b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=ab6b4757201d1008, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:12:44.138 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ab6b4757201d1008,5b6dfbe6d81e411b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=ab6b4757201d1008, duration=1 ms] 2025-10-27 11:12:44.526 INFO 2458 --- [ocp-async-3,7fa85785a10f1110,b92d9892033a53b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:44.526 WARN 2458 --- [ocp-async-7,d6aefd69249f093c,7203d71fed83d6c8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:44.707 INFO 2458 --- [prometheus-minute-schedule-1,83ee54921ea8eafd,ad727d710368cdfe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:44.714 INFO 2458 --- [prometheus-minute-schedule-1,83ee54921ea8eafd,ad727d710368cdfe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:45.093 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:45.094 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:45.102 INFO 2458 --- [prometheus-second-schedule-1,fa5f45eb8a09eda8,204ba2e22993027c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:12:46.088 INFO 2458 --- [ocp-schedules-14,f2e42e4c626db9c3,87a7d251334a36ad] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:46.526 INFO 2458 --- [ocp-async-3,f70cce9f3c97aedb,77c95c55f365eea1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:46.526 WARN 2458 --- [ocp-async-9,2f1c7cf90afe9cea,787457bf15753b96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:46.734 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:12:46.764 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:46.769 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:46.808 INFO 2458 --- [ocp-schedules-5,a34573f26f06722a,41c19a05997c6baa] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:12:46.809 INFO 2458 --- [ocp-schedules-5,a34573f26f06722a,41c19a05997c6baa] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:12:46.816 INFO 2458 --- [ocp-schedules-5,a34573f26f06722a,41c19a05997c6baa] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:12:46.816 INFO 2458 --- [ocp-schedules-5,a34573f26f06722a,41c19a05997c6baa] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:12:46.823 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:46.826 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:46.873 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:46.876 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:46.903 INFO 2458 --- [ocp-schedules-30,bb5ab4a724c4100d,51e05340c3101da1] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:12:46.911 INFO 2458 --- [ocp-schedules-30,bb5ab4a724c4100d,51e05340c3101da1] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:46.911 INFO 2458 --- [ocp-schedules-30,bb5ab4a724c4100d,51e05340c3101da1] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:12:46.929 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:46.933 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:46.988 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:46.991 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:47.042 INFO 2458 --- [ocp-schedules-10,a273b210628535f2,1310dc8c84ae6b9c] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:12:47.707 INFO 2458 --- [prometheus-minute-schedule-1,6016f9eab9769cac,617680e3a1712ea9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:47.714 INFO 2458 --- [prometheus-minute-schedule-1,6016f9eab9769cac,617680e3a1712ea9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:47.889 INFO 2458 --- [ocp-schedules-14,957d4f75d7e14692,c903ef0027ed4b61] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:47.999 WARN 2458 --- [scanner-112149387-1,957d4f75d7e14692,b30ef7d7fb841b1d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:48.015 INFO 2458 --- [alarm-detect-17,957d4f75d7e14692,50e06a997cc85cd9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534697 ~ 1761534757 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534720, value=203054.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203054.56, values={1761534720=203054.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:12:48.059 INFO 2458 --- [alarm-detect-6,957d4f75d7e14692,a2309a66aa6d3746] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534698 ~ 1761534758 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534720, value=93.018)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.018, values={1761534720=93.018}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:12:48.090 INFO 2458 --- [alarm-detect-18,957d4f75d7e14692,fa05445770588d20] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534698 ~ 1761534758 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534720, value=203055.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203055.912, values={1761534720=203055.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:12:48.094 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:48.094 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:48.099 INFO 2458 --- [alarm-detect-14,957d4f75d7e14692,44cb465bc8c6c135] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534735 ~ 1761534758 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534750, value=93.651)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534750, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534750=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:12:48.146 INFO 2458 --- [ocp-schedules-15,903e27e49d011d4a,21d1f203bfcf7cee] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:12:48.146+08:00, clusters=[1, 2] 2025-10-27 11:12:48.163 INFO 2458 --- [ocp-schedules-15,903e27e49d011d4a,21d1f203bfcf7cee] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:12:48.166 INFO 2458 --- [ocp-schedules-15,903e27e49d011d4a,21d1f203bfcf7cee] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:12:48.514 INFO 2458 --- [ocp-async-3,326c06ef0e516d7d,b200587469adb8af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:48.526 WARN 2458 --- [ocp-async-8,c7d465568c158ae2,d65fc0ddaddc2c18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:48.526 INFO 2458 --- [ocp-async-1,e3f061fa1e70225e,22c7d48ee6b0a71c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:50.526 WARN 2458 --- [ocp-async-4,8227d44be68794f1,5ca438e1a6ca71fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:50.526 INFO 2458 --- [ocp-async-2,1d9114276485b17a,12417c46bdf39607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:50.707 INFO 2458 --- [prometheus-minute-schedule-1,7bc5a086b6cd518c,526630b4787b9b8c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:50.715 INFO 2458 --- [prometheus-minute-schedule-1,7bc5a086b6cd518c,526630b4787b9b8c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:51.095 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:51.096 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:51.409 INFO 2458 --- [ocp-schedules-26,c2cd86afeff18c58,e5cb0285845a116c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:51.410 INFO 2458 --- [ocp-schedules-23,cd1fca09f81598d4,74817e9772959d4a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:12:51.410 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:12:51.411 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:12:51.411 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:12:51.411 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:12:51.425 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:51.426 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:51.426 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:51.426 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:51.426 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:51.427 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:51.427 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:51.428 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:51.430 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:51.430 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:51.431 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:51.431 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:51.431 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:51.431 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:51.432 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:51.432 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:51.433 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:51.435 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:51.436 INFO 2458 --- [ocp-schedules-22,6602c2dc09b6b2da,bff61805dfb7b764] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:12:51.436 INFO 2458 --- [host-check-clockdiff1,6602c2dc09b6b2da,3c2149c722cda8df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:12:51.436 INFO 2458 --- [host-check-clockdiff2,6602c2dc09b6b2da,7a487eeb6f937796] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:12:51.437 INFO 2458 --- [host-check-clockdiff3,6602c2dc09b6b2da,936f751dc1453f5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:12:51.437 INFO 2458 --- [host-check-clockdiff4,6602c2dc09b6b2da,366b0d0a2686eaf8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:12:51.437 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:51.438 INFO 2458 --- [host-check-clockdiff5,6602c2dc09b6b2da,d0b67dbfb9102331] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:12:51.440 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:51.442 INFO 2458 --- [ocp-schedules-16,2c7dad2fdefec8ee,b04315685691d481] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:12:51.460 INFO 2458 --- [host-check-clockdiff3,6602c2dc09b6b2da,936f751dc1453f5e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534771 -1 -1 , error= 2025-10-27 11:12:51.463 INFO 2458 --- [host-check-clockdiff2,6602c2dc09b6b2da,7a487eeb6f937796] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534771 -1 -1 , error= 2025-10-27 11:12:51.463 INFO 2458 --- [host-check-clockdiff5,6602c2dc09b6b2da,d0b67dbfb9102331] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534771 0 0 , error= 2025-10-27 11:12:51.465 INFO 2458 --- [host-check-clockdiff1,6602c2dc09b6b2da,3c2149c722cda8df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534771 0 0 , error= 2025-10-27 11:12:51.511 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:12:51.512 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:12:51.513 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:12:51.513 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:12:51.513469777+08:00 2025-10-27 11:12:51.513 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:12:51.514 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:12:51.513+08:00 2025-10-27 11:12:51.514 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:12:51.515 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:12:51.514987497+08:00 2025-10-27 11:12:51.515 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:12:51.513677985+08:00 2025-10-27 11:12:51.515 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:12:51.515+08:00 2025-10-27 11:12:51.516 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:12:51.516 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:12:51.515+08:00 2025-10-27 11:12:51.516 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:12:51.515732287+08:00 2025-10-27 11:12:51.517 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:12:51.516+08:00 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:12:51.516693313+08:00 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-agent-status1,cd1fca09f81598d4,69990307e40801da] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:12:51.518 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:12:51.518+08:00 2025-10-27 11:12:51.520 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:12:51.520 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:12:51.520 INFO 2458 --- [host-check-agent-status5,cd1fca09f81598d4,4d2e77874a4e11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:12:51.523 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:12:51.513+08:00 and status AVAILABLE 2025-10-27 11:12:51.523 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:12:51.523 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:12:51.515+08:00 and status AVAILABLE 2025-10-27 11:12:51.524 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:12:51.524 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:12:51.515+08:00 and status AVAILABLE 2025-10-27 11:12:51.525 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:12:51.525 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:12:51.525 INFO 2458 --- [host-check-agent-status4,cd1fca09f81598d4,4fb412bcf81ab06a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:12:51.526 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:12:51.516+08:00 and status AVAILABLE 2025-10-27 11:12:51.527 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:12:51.518+08:00 and status AVAILABLE 2025-10-27 11:12:51.527 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:12:51.529 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:12:51.529 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:12:51.530 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:12:51.530 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:12:51.530 INFO 2458 --- [host-check-agent-status3,cd1fca09f81598d4,85d6af013a25b4fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:12:51.531 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:12:51.533 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:12:51.534 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:12:51.537 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:12:51.537 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:12:51.537 INFO 2458 --- [host-check-agent-status2,cd1fca09f81598d4,9f31622e60c2e915] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:12:51.545 INFO 2458 --- [ocp-schedules-19,42096e33387c603c,ff492aa6db145dbb] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:12:51.549 INFO 2458 --- [ocp-schedules-18,066937b3c2d108d9,d2319f467075945f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:12:51.549 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:12:51.550 INFO 2458 --- [ocp-schedules-19,42096e33387c603c,ff492aa6db145dbb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:51.552 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:12:51.552 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:51.553 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:12:51.553 INFO 2458 --- [ocp-schedules-18,066937b3c2d108d9,d2319f467075945f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:51.554 INFO 2458 --- [ocp-schedules-21,b4d982cb0190df13,a891a91189306b7d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:12:51.554 INFO 2458 --- [ocp-schedules-18,066937b3c2d108d9,d2319f467075945f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:12:51.554 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:12:51.555 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:51.555 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:12:51.556 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:51.557 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:12:51.558 INFO 2458 --- [ocp-schedules-21,b4d982cb0190df13,a891a91189306b7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:51.559 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:51.566 INFO 2458 --- [ocp-schedules-10,523cc4e2b530488e,73749579920a7cb7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:12:51.568 INFO 2458 --- [ocp-schedules-15,8e835c0d38243aa1,9d2a6d2568c3a889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:12:51.569 INFO 2458 --- [ocp-schedules-18,c9ae6464240cab70,4a39147c4ada8b29] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:12:51.569 INFO 2458 --- [ocp-schedules-3,29d5a54223fd95de,f6ae3bb18c1e620e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:12:51.572 INFO 2458 --- [ocp-schedules-10,523cc4e2b530488e,73749579920a7cb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:12:51.577 INFO 2458 --- [ocp-schedules-15,8e835c0d38243aa1,9d2a6d2568c3a889] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:12:51.579 INFO 2458 --- [ocp-schedules-3,29d5a54223fd95de,f6ae3bb18c1e620e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:12:51.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.595 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:51.596 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:12:51.613 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:51.615 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.615 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:12:51.618 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.620 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:51.620 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:51.624 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6c6cfaaa150ccdbd,2ebd23a8631acca5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=6c6cfaaa150ccdbd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:12:51.624 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,6c6cfaaa150ccdbd,2ebd23a8631acca5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=6c6cfaaa150ccdbd, duration=1 ms] 2025-10-27 11:12:51.629 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.629 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:12:51.630 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:51.635 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:12:51.635 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:12:51.636 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:12:51.638 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.640 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:12:51.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.640 INFO 2458 --- [ocp-schedules-19,42096e33387c603c,ff492aa6db145dbb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:12:51.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:51.642 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:12:51.642 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.642 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.643 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.643 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.643 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.645 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:12:51.648 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:12:51.650 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:12:51.660 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:12:51.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,c7ffb3d341083111,d160ce9b89437bb9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=c7ffb3d341083111, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:12:51.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,c7ffb3d341083111,d160ce9b89437bb9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=c7ffb3d341083111, duration=1 ms] 2025-10-27 11:12:51.707 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.707 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.708 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.709 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:12:51.711 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:12:51.724 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:12:51.727 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.727 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.728 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:12:51.728 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.731 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:12:51.731 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:12:51.729782+08:00, lastProcessTime=2025-10-27T11:12:41.716521+08:00 2025-10-27 11:12:51.732 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.732 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.732 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:12:51.732 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:51.732 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.733 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.734 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:12:51.735 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:12:51.735 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.735 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.737 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.738 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:12:51.740 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:12:51.741 INFO 2458 --- [ocp-schedules-10,523cc4e2b530488e,73749579920a7cb7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:12:51.746 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.746 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.747 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.747 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.747 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.750 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:51.784 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:51.784 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:51.785 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:51.787 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:51.788 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:12:51.800 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:12:51.802 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:12:51.806 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:12:51.808 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:12:51.817 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:51.818 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.819 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:51.820 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:12:51.820 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:12:51.825 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:51.838 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:12:51.839 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:12:51.853 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:12:51.856 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:51.856 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:51.856 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:51.856 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.856 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.857 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.857 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.857 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.858 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:12:51.860 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:12:51.861 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:12:51.878 INFO 2458 --- [ocp-schedules-30,3768aa2670dc1c21,59694218d4b3c41c] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:12:51.878 INFO 2458 --- [ocp-schedules-30,3768aa2670dc1c21,59694218d4b3c41c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:12:51.879 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:51.881 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:51.892 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:51.896 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:12:51.896 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:51.897 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:12:51.898 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:51.904 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:12:51.916 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:12:51.917 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:12:51.928 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:51.929 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.930 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:51.931 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:12:51.933 INFO 2458 --- [host-check-host-status5,c2cd86afeff18c58,56b8ae79c81f3c5a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:12:51.939 INFO 2458 --- [host-check-host-status4,c2cd86afeff18c58,056853d843fd1d52] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:12:51.939 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:12:51.950 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:12:51.955 INFO 2458 --- [host-check-host-status3,c2cd86afeff18c58,f729741fbd37ac28] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:12:51.960 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:12:51.963 INFO 2458 --- [host-check-host-status2,c2cd86afeff18c58,e3a88d474a0ccbe4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:12:51.965 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:12:51.968 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:51.969 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.969 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:12:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:51.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:12:51.979 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:51.985 INFO 2458 --- [host-check-host-status1,c2cd86afeff18c58,5e1beeca69b0d3f5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:12:52.017 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:12:51.729782+08:00 2025-10-27 11:12:52.022 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:12:52.022 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.022 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:52.060 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.060 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:52.064 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:12:52.065 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.065 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:12:52.067 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:12:52.084 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:12:52.093 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.093 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:52.095 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.100 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, 2025-10-27 11:12:52.107 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.108 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.109 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:12:52.116 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.117 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:12:52.122 INFO 2458 --- [pool-overview-container-7,fbe81ae2075fb13c,9d109fc17570f902] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.127 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.131 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:12:52.131 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:12:52.131 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.131 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:12:52.138 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 7 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:12:52.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:12:52.238 INFO 2458 --- [pool-overview-container-1,a2f4c5e817c937ba,4e95a9fa2bed6a03] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.262 INFO 2458 --- [pool-overview-container-5,66558dd7ac765c6e,3ebb1624a3e95741] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.265 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.266 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:12:52.267 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.267 INFO 2458 --- [pool-overview-container-5,66558dd7ac765c6e,3ebb1624a3e95741] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.268 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.269 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:12:52.269 INFO 2458 --- [pool-overview-container-5,66558dd7ac765c6e,3ebb1624a3e95741] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:12:52.270 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.271 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:12:52.271 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.272 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.274 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.280 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:12:52.284 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.285 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.286 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.288 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.289 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.291 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:12:52.294 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:12:52.296 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.297 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.298 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.299 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.301 INFO 2458 --- [pool-12-thread-8,a2f4c5e817c937ba,3ffcd212fe4ead84] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:12:52.302 INFO 2458 --- [pool-12-thread-6,a2f4c5e817c937ba,7693f85c6d8e5d9e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:12:52.302 INFO 2458 --- [pool-12-thread-5,a2f4c5e817c937ba,62da60716de31837] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:12:52.303 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.306 INFO 2458 --- [pool-12-thread-7,a2f4c5e817c937ba,a01488743d2b3831] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:12:52.306 INFO 2458 --- [pool-12-thread-7,a2f4c5e817c937ba,a01488743d2b3831] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:12:52.307 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.309 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.312 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.313 INFO 2458 --- [ForkJoinPool-7-worker-10,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.314 INFO 2458 --- [ForkJoinPool-7-worker-12,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.314 INFO 2458 --- [ForkJoinPool-7-worker-5,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.315 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:12:52.315 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:12:52.315 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:12:52.315 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:12:52.320 INFO 2458 --- [ForkJoinPool-7-worker-3,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:12:52.322 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.324 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.326 INFO 2458 --- [pool-12-thread-5,a2f4c5e817c937ba,62da60716de31837] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:12:52.326 INFO 2458 --- [pool-12-thread-5,a2f4c5e817c937ba,62da60716de31837] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:12:52.335 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.336 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.337 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.342 INFO 2458 --- [pool-12-thread-7,a2f4c5e817c937ba,a01488743d2b3831] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:12:52.344 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:06:51.812, end=2025-10-27T11:11:52.087} 2025-10-27 11:12:52.344 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 1 dubious sqls 2025-10-27 11:12:52.348 INFO 2458 --- [pool-12-thread-7,a2f4c5e817c937ba,a01488743d2b3831] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:12:52.358 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.360 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:12:52.361 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:12:52.363 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.363 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:12:52.364 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.365 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:12:52.372 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.372 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.373 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.373 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.373 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.373 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.373 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.374 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.374 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.374 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.374 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:12:52.376 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:12:52.383 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.383 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.383 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.385 INFO 2458 --- [host-check-clockdiff4,6602c2dc09b6b2da,366b0d0a2686eaf8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534772 0 0 , error= 2025-10-27 11:12:52.386 INFO 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:12:52.393 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:12:52.393 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.394 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:12:52.395 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.396 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:12:52.409 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.409 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:12:52.410 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.411 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:12:52.411 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:12:52.415 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:12:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.417 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:12:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.420 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:12:52.428 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.433 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:12:52.436 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:12:52.436 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.436 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.436 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.436 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.437 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.438 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:12:52.439 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.441 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:12:52.442 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.442 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.442 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.444 INFO 2458 --- [ForkJoinPool-7-worker-3,a2f4c5e817c937ba,f2c1cd1642357167] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:12:52.449 WARN 2458 --- [ForkJoinPool-7-worker-5,a2f4c5e817c937ba,fa135121595b619a] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:12:52.450 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:12:52.456 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.458 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:12:52, 2025-10-27 11:12:52] 2025-10-27 11:12:52.465 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:12:52.477 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:12:52.478 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.479 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:12:52.480 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.481 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.481 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.481 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.481 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.481 INFO 2458 --- [ForkJoinPool-7-worker-10,a2f4c5e817c937ba,fcd426d37ebe543d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:12:52.481 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.483 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:12:52.483 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.485 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:12:52, 2025-10-27 11:12:52] 2025-10-27 11:12:52.486 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:12:52.487 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:12:52.487 INFO 2458 --- [ForkJoinPool-7-worker-12,a2f4c5e817c937ba,2d9f87aee24ae3a1] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:12:52.487 INFO 2458 --- [pool-12-thread-6,a2f4c5e817c937ba,7693f85c6d8e5d9e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:12:52.495 INFO 2458 --- [pool-12-thread-8,a2f4c5e817c937ba,3ffcd212fe4ead84] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.509 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.510 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:12:52.514 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.518 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:12:52.526 WARN 2458 --- [ocp-async-10,fde7b944f82a7e35,9793b20ed4e7d3b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:52.526 INFO 2458 --- [ocp-async-8,acaa4761bcac1b41,a8845df2fbca84c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:52.551 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:12:52.551 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:12:52.551 INFO 2458 --- [ocp-schedules-21,b4d982cb0190df13,a891a91189306b7d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:12:52.605 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:12:52.605 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.605 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.606 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:12:52.606 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:12:52.606 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:12:52.610 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:12:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:12:53.707 INFO 2458 --- [prometheus-minute-schedule-1,e56a4c38a9ae8181,5209c12078ecfa18] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:53.715 INFO 2458 --- [prometheus-minute-schedule-1,e56a4c38a9ae8181,5209c12078ecfa18] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:53.971 INFO 2458 --- [ocp-schedules-28,af2c9dc325cd9b46,b1584ccd58d0e417] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:12:53.975 INFO 2458 --- [ocp-schedules-28,af2c9dc325cd9b46,b1584ccd58d0e417] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:12:54.096 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:54.097 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:12:54.526 INFO 2458 --- [ocp-async-7,89ea1a4a431bfaa0,5148f0e86ac90aa8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:54.526 WARN 2458 --- [ocp-async-4,2c368fa35ed08e3a,5a788b38e45d10d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:54.660 INFO 2458 --- [ocp-schedules-11,f0b800585c432ffa,3c8e200015dc8411] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:12:56.096 INFO 2458 --- [ocp-schedules-30,fed018ad057e4435,0e474f52ee48c074] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:12:56.526 WARN 2458 --- [ocp-async-9,5151ec1989bcd5e4,46c74d68ddb1ffa7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:56.526 INFO 2458 --- [ocp-async-9,8838cc27518c50f3,938829c71adda251] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:56.707 INFO 2458 --- [prometheus-minute-schedule-1,a1c4240f807bec9d,f2942fd1226bd679] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:56.715 INFO 2458 --- [prometheus-minute-schedule-1,a1c4240f807bec9d,f2942fd1226bd679] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:12:57.097 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:12:57.098 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:12:58.105 INFO 2458 --- [ocp-schedules-24,3e3eeda3ff0e0bd7,0d80e886fed99dcd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:12:58.129 WARN 2458 --- [scanner-112149641-1,3e3eeda3ff0e0bd7,bd8c1fd378957071] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:12:58.277 INFO 2458 --- [alarm-detect-4,3e3eeda3ff0e0bd7,ced214684026caf5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534750 ~ 1761534768 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534765, value=75.224)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534765, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534765=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:12:58.514 INFO 2458 --- [ocp-async-6,94a15461b71712a2,368e27ecb755fa04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:12:58.526 WARN 2458 --- [ocp-async-5,a459f8fc9dfa09f1,68d231b5522d0d08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:12:58.527 INFO 2458 --- [ocp-async-8,1909424749c80fe1,694e048529554cc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:12:59.254 INFO 2458 --- [ocp-schedules-27,63c866da74edd0b3,2dc5378c38fe6d12] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:12:59.587 INFO 2458 --- [ocp-schedules-4,796b60ddca620d2c,6d1372e246fe1d15] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:12:59.592 INFO 2458 --- [ocp-schedules-4,796b60ddca620d2c,6d1372e246fe1d15] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:12:59.707 INFO 2458 --- [prometheus-minute-schedule-1,46b814b1a8e9d4ca,3a506a4b8fed64ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534780, interval=60 2025-10-27 11:12:59.715 INFO 2458 --- [prometheus-minute-schedule-1,46b814b1a8e9d4ca,3a506a4b8fed64ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:00.099 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:00.099 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:00.527 WARN 2458 --- [ocp-async-4,e8043ce16cd048ac,90d94fa961203995] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:00.527 INFO 2458 --- [ocp-async-7,b9eb14fd8c99c9b5,0286d7075bc9f4e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:00.703 INFO 2458 --- [ocp-schedules-29,f5fed906985c969d,b1c94d90a283ef7b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:13:00.719 INFO 2458 --- [ocp-schedules-29,f5fed906985c969d,b1c94d90a283ef7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:00.836 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:13:00.836 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.840 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:13:00.852 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.853 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.885 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.886 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.893 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.896 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:13:00.916 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.916 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.916 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.917 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.917 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.920 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.920 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:00.949 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.950 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:00.951 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.951 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.952 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.978 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.978 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:00.985 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.985 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.986 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.987 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:13:00.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:00.989 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:00.989 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:00.990 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:01.016 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.018 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.018 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.019 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.020 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.021 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.021 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.021 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.021 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.022 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.022 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:01.023 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.023 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.024 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.025 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.026 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.026 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:13:01.029 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:13:01.055 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.055 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.055 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.056 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:01.058 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.058 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.058 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.059 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:13:01.084 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:01.085 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.085 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:13:01.091 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.091 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.091 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.092 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:13:01.095 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.095 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:13:01.095 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.095 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:13:01.098 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:13:01.098 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:13:01.128 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.128 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.128 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.129 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:13:01.131 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.131 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.132 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.132 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:13:01.166 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.166 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.167 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.167 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:13:01.196 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.196 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:13:01.245 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.245 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.245 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.246 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.247 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.247 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.247 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.248 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.248 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.248 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.248 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.249 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.249 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.249 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.251 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.251 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.253 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.254 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.256 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.256 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.258 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.258 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.260 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.260 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:13:01.261 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.261 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:13:01.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:13:01.264 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:13:01.267 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:13:01.364 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:01.364 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.364 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.365 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:01.365 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:01.365 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:01.366 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:13:01.394 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:13:01.395 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:13:01.399 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:13:01.399 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:13:01.399 INFO 2458 --- [ocp-schedules-29,f5fed906985c969d,b1c94d90a283ef7b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:13:01.729 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:01.727847+08:00, lastProcessTime=2025-10-27T11:12:51.729782+08:00 2025-10-27 11:13:01.959 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:01.727847+08:00 2025-10-27 11:13:02.527 WARN 2458 --- [ocp-async-2,8311ed881486f22a,ace4e21a4a7328e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:02.527 INFO 2458 --- [ocp-async-9,ac5ac9f920602f05,ed462c14025a8aa3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:02.608 INFO 2458 --- [ocp-schedules-19,25082e3c68c7c103,a85179ff4a52be06] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:13:02.617 INFO 2458 --- [ocp-schedules-19,25082e3c68c7c103,a85179ff4a52be06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:02.668 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:02.668 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:13:02.688 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:02.688 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:02.688 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:02.688 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:02.689 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:02.689 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:13:02.707 INFO 2458 --- [ocp-schedules-19,25082e3c68c7c103,a85179ff4a52be06] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:13:02.707 INFO 2458 --- [prometheus-minute-schedule-1,11632d445c92d25b,02491efed6a82fbe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:02.715 INFO 2458 --- [prometheus-minute-schedule-1,11632d445c92d25b,02491efed6a82fbe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:03.099 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:03.100 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:03.983 INFO 2458 --- [ocp-schedules-10,6a7e24176090b8f1,1a9c20866bb7c6cf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:03.987 INFO 2458 --- [ocp-schedules-10,6a7e24176090b8f1,1a9c20866bb7c6cf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:04.527 WARN 2458 --- [ocp-async-10,b1eba014165e857b,6625e2fd65a8e200] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:04.527 INFO 2458 --- [ocp-async-10,1f9ec26bb17e8bee,cc0b0f970336f328] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:05.707 INFO 2458 --- [prometheus-minute-schedule-1,e68c2fe506932fe8,9db41fc635316967] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:05.715 INFO 2458 --- [prometheus-minute-schedule-1,e68c2fe506932fe8,9db41fc635316967] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:06.100 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:06.102 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:06.103 INFO 2458 --- [ocp-schedules-5,a29c79e6005e9db7,cd2e56642c133b9e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:06.527 WARN 2458 --- [ocp-async-4,4a9fa1d521dbc81d,123017067c27d649] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:06.527 INFO 2458 --- [ocp-async-4,d25f5c5eb580d5aa,4de9e135e0fe89b7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:06.913 INFO 2458 --- [ocp-schedules-17,793cd7efd7882d00,9c8c690247ebb147] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:13:06.924 INFO 2458 --- [ocp-schedules-17,793cd7efd7882d00,9c8c690247ebb147] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:06.924 INFO 2458 --- [ocp-schedules-17,793cd7efd7882d00,9c8c690247ebb147] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:07.112 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9ee8cdb00714a63d,b527106e3d6e4bf6] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/tasks/instances/9096/rollback, client 10.40.198.13, traceId 9ee8cdb00714a63d] 2025-10-27 11:13:07.115 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9ee8cdb00714a63d,b527106e3d6e4bf6] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/tasks/instances/9096/rollback?taskInstanceId=9096, client=10.40.198.13, traceId=9ee8cdb00714a63d, method=NoDataResponse com.oceanbase.ocp.server.common.controller.TaskController.rollbackTask(Long), args=9096,] 2025-10-27 11:13:07.138 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9ee8cdb00714a63d,b527106e3d6e4bf6] c.o.o.c.m.r.t.m.SubtaskInstanceEntity : Set state for subtask: 9149, current state: FAILED, new state: READY 2025-10-27 11:13:07.163 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,9ee8cdb00714a63d,b527106e3d6e4bf6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /api/v2/tasks/instances/9096/rollback client=10.40.198.13, traceId=9ee8cdb00714a63d, duration=72 ms] 2025-10-27 11:13:07.270 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,d2fc058a3af7b89a,a9468f375053443c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=d2fc058a3af7b89a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:13:07.283 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,ba9b093816d9f599,7a198ea6a774e921] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=ba9b093816d9f599, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:13:07.285 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,ba9b093816d9f599,7a198ea6a774e921] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=ba9b093816d9f599, duration=12 ms] 2025-10-27 11:13:07.297 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,d2fc058a3af7b89a,a9468f375053443c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=d2fc058a3af7b89a, duration=35 ms] 2025-10-27 11:13:07.535 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,ae87f3b50fe6963f,8f783c7d15aa9d42] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments?taskInstanceId=9096&subtaskInstanceId=9149, client=10.40.198.13, traceId=ae87f3b50fe6963f, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9096,9149,] 2025-10-27 11:13:07.548 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,ae87f3b50fe6963f,8f783c7d15aa9d42] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments client=10.40.198.13, traceId=ae87f3b50fe6963f, duration=22 ms] 2025-10-27 11:13:07.563 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,add3a487330cc648] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=9149, original=READY 2025-10-27 11:13:07.569 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,add3a487330cc648] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9149, operation:ROLLBACK, state: ROLLING_BACK 2025-10-27 11:13:07.605 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:13:07.612 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=9149, context=Context{parallelIdx=-1, stringMap={tenant_id=6, target_tenant_binlog_service_status=RUNNING, cluster_cdcro_password=xxx task_instance_id=9096, task_operation=rollback, binlog_cluster_id=4, binlog_instance_replicate_count=1, tenant_binlog_service_id=9, operation_scope=Tenant, latest_execution_start_time=2025-10-27T11:13:07.563+08:00, sub_task_instance_name=Create tenant binlog service, sub_task_instance_id=9149, cluster_cdcro_username=cdcro}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, operation=ROLLBACK, executor=10.40.65.146 2025-10-27 11:13:07.620 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:13:07.653 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES FOR `myoceanbase2`.`test2` 2025-10-27 11:13:07.791 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9149, operation:ROLLBACK, state: PENDING 2025-10-27 11:13:07.801 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,61135fcc2064a911] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9147] 2025-10-27 11:13:08.283 INFO 2458 --- [ocp-schedules-27,442c86b242ff9e54,ff45078778e16cd1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:08.350 WARN 2458 --- [scanner-112149824-1,442c86b242ff9e54,dee813892d55ec32] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:08.477 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,cf96855adfaf7772,2990c483d7c87bd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=cf96855adfaf7772, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:13:08.504 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,cf96855adfaf7772,2990c483d7c87bd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=cf96855adfaf7772, duration=35 ms] 2025-10-27 11:13:08.514 INFO 2458 --- [ocp-async-2,3ca836055d5e7b52,9b48f663fdd3aa59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:08.527 WARN 2458 --- [ocp-async-1,794a29065a514781,a5b9ecb5ec064c56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:08.527 INFO 2458 --- [ocp-async-5,79e35eeb31e50f46,15378e0bfefdc2d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:08.567 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,14996863f4872a63,e400949612fd0520] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments?taskInstanceId=9096&subtaskInstanceId=9149, client=10.40.198.13, traceId=14996863f4872a63, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9096,9149,] 2025-10-27 11:13:08.569 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,14996863f4872a63,e400949612fd0520] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096/subtasks/9149/logSegments client=10.40.198.13, traceId=14996863f4872a63, duration=9 ms] 2025-10-27 11:13:08.586 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,902ed7b209056682] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=9147, original=READY 2025-10-27 11:13:08.588 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,902ed7b209056682] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9147, operation:ROLLBACK, state: ROLLING_BACK 2025-10-27 11:13:08.612 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,1c1636bcb369274b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:13:08.618 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,1c1636bcb369274b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=9147, context=Context{parallelIdx=-1, stringMap={tenant_id=6, target_tenant_binlog_service_status=RUNNING, cluster_cdcro_password=xxx task_instance_id=9096, task_operation=rollback, binlog_cluster_id=4, binlog_instance_replicate_count=1, tenant_binlog_service_id=9, operation_scope=Tenant, latest_execution_start_time=2025-10-27T11:13:08.586+08:00, sub_task_instance_name=Create cluster cdcro user, sub_task_instance_id=9147, cluster_cdcro_username=cdcro}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, operation=ROLLBACK, executor=10.40.65.146 2025-10-27 11:13:08.630 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,1c1636bcb369274b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9147, operation:ROLLBACK, state: PENDING 2025-10-27 11:13:08.632 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,1c1636bcb369274b] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9146] 2025-10-27 11:13:08.707 INFO 2458 --- [prometheus-minute-schedule-1,6d45db83f7f94a25,4f94b9b9f4a97e19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:08.714 INFO 2458 --- [prometheus-minute-schedule-1,6d45db83f7f94a25,4f94b9b9f4a97e19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:09.102 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:09.103 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:09.569 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,0a7a45eee795225f,4c2d6bba6fe92c62] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=0a7a45eee795225f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:13:09.590 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,0a7a45eee795225f,4c2d6bba6fe92c62] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=0a7a45eee795225f, duration=29 ms] 2025-10-27 11:13:09.599 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,6b271c6a08cc4f61] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set subtask state to ROLLING_BACK, id=9146, original=READY 2025-10-27 11:13:09.601 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,6b271c6a08cc4f61] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9146, operation:ROLLBACK, state: ROLLING_BACK 2025-10-27 11:13:09.628 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,bc59687b90499a9b] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:13:09.634 WARN 2458 --- [manual-subtask-executor16,c791b4f95d069192,bc59687b90499a9b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Rollback subtask, id=9146, context=Context{parallelIdx=-1, stringMap={binlog_instance_replicate_count=1, tenant_id=6, tenant_binlog_service_id=9, target_tenant_binlog_service_status=RUNNING, operation_scope=Tenant, task_instance_id=9096, task_operation=rollback, latest_execution_start_time=2025-10-27T11:13:09.599+08:00, sub_task_instance_name=Prepare create tenant binlog service, sub_task_instance_id=9146, binlog_cluster_id=4}, listMap={binlog_service_obproxy_cluster_ids_to_add=[1], binlog_instance_parameters=[binlog_expire_logs_seconds|3600, binlog_expire_logs_size|1073741824]}}, operation=ROLLBACK, executor=10.40.65.146 2025-10-27 11:13:09.704 INFO 2458 --- [manual-subtask-executor16,c791b4f95d069192,bc59687b90499a9b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9146, operation:ROLLBACK, state: PENDING 2025-10-27 11:13:10.013 INFO 2458 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:13:10.013 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:13:10.048 INFO 2458 --- [Check obproxy1,b53762472cb4f17d,715ec8b1438cd215] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:13:10.048 INFO 2458 --- [Check obproxy1,b53762472cb4f17d,715ec8b1438cd215] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:13:10.049 INFO 2458 --- [Collect obproxy1,32bf1ab6f17c63fb,4199b4d53f0a6cb3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:13:10.049 INFO 2458 --- [Collect obproxy1,32bf1ab6f17c63fb,4199b4d53f0a6cb3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:13:10.097 INFO 2458 --- [Check obproxy1,b53762472cb4f17d,715ec8b1438cd215] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:13:10.097 INFO 2458 --- [Collect obproxy1,32bf1ab6f17c63fb,4199b4d53f0a6cb3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:13:10.100 INFO 2458 --- [Check obproxy1,b53762472cb4f17d,715ec8b1438cd215] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:13:10.100 INFO 2458 --- [Check obproxy1,b53762472cb4f17d,715ec8b1438cd215] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:13:10.101 INFO 2458 --- [Collect obproxy1,32bf1ab6f17c63fb,4199b4d53f0a6cb3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:13:10.103 INFO 2458 --- [Collect obproxy1,32bf1ab6f17c63fb,4199b4d53f0a6cb3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:13:10.179 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:13:10.527 WARN 2458 --- [ocp-async-8,4583830dbeef1928,523caa35a28262c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:10.527 INFO 2458 --- [ocp-async-6,1594de2df47c5687,9a57d0814b4154de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:10.619 WARN 2458 --- [task-coordinator1,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm type ob_tenant_operation_info is not enabled 2025-10-27 11:13:10.620 INFO 2458 --- [task-coordinator1,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"status":"Disabled"} 2025-10-27 11:13:10.653 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=296, affectRows=296 2025-10-27 11:13:10.668 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,356539bef6e3ec2f,d3e120c3b5a1f99f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=356539bef6e3ec2f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:13:10.697 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=101, affectRows=101 2025-10-27 11:13:10.698 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,356539bef6e3ec2f,d3e120c3b5a1f99f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=356539bef6e3ec2f, duration=39 ms] 2025-10-27 11:13:10.731 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=99, affectRows=99 2025-10-27 11:13:10.764 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=100, affectRows=100 2025-10-27 11:13:10.803 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=144, affectRows=144 2025-10-27 11:13:10.844 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534720, nodeCount=81, affectRows=81 2025-10-27 11:13:11.632 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,0f59afdc5ad52bfd,8c98f21d8e226be9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9096?taskInstanceId=9096, client=10.40.198.13, traceId=0f59afdc5ad52bfd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9096,] 2025-10-27 11:13:11.652 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,0f59afdc5ad52bfd,8c98f21d8e226be9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9096 client=10.40.198.13, traceId=0f59afdc5ad52bfd, duration=32 ms] 2025-10-27 11:13:11.707 INFO 2458 --- [prometheus-minute-schedule-1,fb14fa4749aa95ce,a319b5d1aacbf593] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:11.713032+08:00, lastProcessTime=2025-10-27T11:13:01.727847+08:00 2025-10-27 11:13:11.715 INFO 2458 --- [prometheus-minute-schedule-1,fb14fa4749aa95ce,a319b5d1aacbf593] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:11.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:11.713032+08:00 2025-10-27 11:13:12.103 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:12.103 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:12.527 WARN 2458 --- [ocp-async-7,d06e320321d4d7d4,dc3426f10b513ab1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:12.527 INFO 2458 --- [ocp-async-7,7b44e717a624f551,7b7206a0c1c1b7d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:13.994 INFO 2458 --- [ocp-schedules-11,31c578a49f54fa4a,f0ee6916338111a2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:13.998 INFO 2458 --- [ocp-schedules-11,31c578a49f54fa4a,f0ee6916338111a2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:14.091 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=1000, affectRows=1000 2025-10-27 11:13:14.095 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=994, validCount=992 2025-10-27 11:13:14.322 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=1000, affectRows=1000 2025-10-27 11:13:14.326 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=999, validCount=995 2025-10-27 11:13:14.376 ERROR 2458 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538020) Duplicate entry '835370602992181248-1761534660' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538020) Duplicate entry '835370602992181248-1761534660' for key 'PRIMARY' 2025-10-27 11:13:14.528 WARN 2458 --- [ocp-async-9,09517fdcdc52e2a8,6b9ae725a3366661] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:14.528 INFO 2458 --- [ocp-async-2,a61afe345301af49,def79ff54a25af49] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:14.707 INFO 2458 --- [prometheus-minute-schedule-1,e7a5b7aac1ce22b4,32c9316c72871a82] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:14.714 INFO 2458 --- [prometheus-minute-schedule-1,e7a5b7aac1ce22b4,32c9316c72871a82] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:14.728 INFO 2458 --- [ocp-schedules-25,1c1dfb0e3dff70f6,69e8580a198301a8] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:13:14.728+08:00, clusters=[1, 2] 2025-10-27 11:13:14.729 INFO 2458 --- [ocp-schedules-25,1c1dfb0e3dff70f6,69e8580a198301a8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:13:14.730 INFO 2458 --- [ocp-schedules-25,1c1dfb0e3dff70f6,69e8580a198301a8] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:13:15.104 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:15.106 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:15.126 INFO 2458 --- [ocp-schedules-27,93a60f074459ab9a,3afbc0d0387165e8] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:13:15.126 INFO 2458 --- [ocp-schedules-27,93a60f074459ab9a,3afbc0d0387165e8] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:13:15.191 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:13:15.192 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:13:15.192 INFO 2458 --- [Collect all tenants major compactions6,93a60f074459ab9a,f8ae609b39698a60] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:13:15.192 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:13:15.192 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:13:15.192 INFO 2458 --- [Collect all tenants major compactions5,93a60f074459ab9a,3bcdbceb4e380e26] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:13:15.225 INFO 2458 --- [Collect all tenants major compactions5,93a60f074459ab9a,3bcdbceb4e380e26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.227 INFO 2458 --- [Collect all tenants major compactions5,93a60f074459ab9a,3bcdbceb4e380e26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:13:15.229 INFO 2458 --- [Collect all tenants major compactions6,93a60f074459ab9a,f8ae609b39698a60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.231 INFO 2458 --- [Collect all tenants major compactions6,93a60f074459ab9a,f8ae609b39698a60] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:13:15.231 INFO 2458 --- [Collect all tenants major compactions5,93a60f074459ab9a,3bcdbceb4e380e26] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:13:15.234 INFO 2458 --- [Collect all tenants major compactions6,93a60f074459ab9a,f8ae609b39698a60] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:13:15.243 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.243 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.243 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.243 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.244 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.245 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:15.246 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:15.247 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:13:15.247 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:13:15.249 INFO 2458 --- [Collect all tenants major compactions1,93a60f074459ab9a,f83ba311774f57e0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:13:15.249 INFO 2458 --- [Collect all tenants major compactions3,93a60f074459ab9a,b7d7081fe780d10a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:13:15.250 INFO 2458 --- [Collect all tenants major compactions4,93a60f074459ab9a,ba46ab1f314d5b70] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:13:15.251 INFO 2458 --- [Collect all tenants major compactions2,93a60f074459ab9a,498c540a2fba2f53] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:13:15.252 INFO 2458 --- [ocp-schedules-27,93a60f074459ab9a,3afbc0d0387165e8] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:13:15.255 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835014129065861120-1761534660' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835014129065861120-1761534660' for key 'PRIMARY' 2025-10-27 11:13:15.716 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=77, affectRows=77 2025-10-27 11:13:15.716 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=76, validCount=76 2025-10-27 11:13:15.859 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=1000, affectRows=1000 2025-10-27 11:13:15.863 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=996, validCount=992 2025-10-27 11:13:16.110 INFO 2458 --- [ocp-schedules-28,cd7997f2bf8acab5,3197e505eb80d2b5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:16.528 WARN 2458 --- [ocp-async-5,efcb04e563728bdd,383f96beb22e60ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:16.528 INFO 2458 --- [ocp-async-10,260b06547315aca7,b0b25eda6b8ee1bd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:16.963 INFO 2458 --- [ocp-schedules-4,30c1e63c6138d1f3,b28fb1d6961224f3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:13:16.964 INFO 2458 --- [ocp-schedules-4,30c1e63c6138d1f3,b28fb1d6961224f3] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:13:16.971 INFO 2458 --- [ocp-schedules-4,30c1e63c6138d1f3,b28fb1d6961224f3] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:13:16.971 INFO 2458 --- [ocp-schedules-4,30c1e63c6138d1f3,b28fb1d6961224f3] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:13:17.707 INFO 2458 --- [prometheus-minute-schedule-1,87314fbc89cdc3b1,bd9401f9b8642f07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:17.715 INFO 2458 --- [prometheus-minute-schedule-1,87314fbc89cdc3b1,bd9401f9b8642f07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:18.106 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:18.106 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:18.515 INFO 2458 --- [ocp-async-6,9e4e58bf88965d4a,0eba063cb0ee3a3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:18.526 INFO 2458 --- [ocp-schedules-4,32316e1cd77f2993,441de13f63f4f76f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:18.528 WARN 2458 --- [ocp-async-3,afd876b23f61e08e,64db676f0cd987ac] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:18.528 INFO 2458 --- [ocp-async-7,11176a148da56f2e,f0b761013f2d6c68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:18.551 WARN 2458 --- [scanner-112151620-1,32316e1cd77f2993,29b2c5cc1e34fa06] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:18.643 INFO 2458 --- [alarm-detect-7,32316e1cd77f2993,d2e7160430d0698e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534765 ~ 1761534788 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=68.21)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534780=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:13:19.513 INFO 2458 --- [ocp-schedules-12,8e71e292a08f3ced,db59bd10990ccfce] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:13:20.528 WARN 2458 --- [ocp-async-1,1744afcce22f26b8,59c2a3273d0f5e87] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:20.528 INFO 2458 --- [ocp-async-2,63b95db0ab4e5d1a,b057015b55801693] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:20.707 INFO 2458 --- [prometheus-minute-schedule-1,68a2b8d1d74f3101,514616e78a76b0bc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:20.716 INFO 2458 --- [prometheus-minute-schedule-1,68a2b8d1d74f3101,514616e78a76b0bc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:21.107 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:21.107 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:21.409 INFO 2458 --- [ocp-schedules-28,1531e329982b01cb,bfbfbb85b7ae7c5c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:21.410 INFO 2458 --- [ocp-schedules-24,44687dcd8417d64e,5b45c2fe0fd032d5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:21.410 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:13:21.410 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:13:21.410 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:13:21.411 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:13:21.411 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:13:21.412 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:13:21.412 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:13:21.412 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:13:21.413 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:13:21.413 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:13:21.426 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:21.426 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:21.427 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:21.427 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:21.427 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:21.427 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:21.428 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:21.431 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:21.431 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:21.431 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:21.431 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:21.432 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:21.432 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:21.433 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:21.433 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:21.434 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:21.434 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:21.434 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:21.436 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:21.437 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:21.438 INFO 2458 --- [ocp-schedules-9,feed99a36f42730f,a6f13ab3a4de4455] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:21.439 INFO 2458 --- [host-check-clockdiff1,feed99a36f42730f,85f9822ac9a75efd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:13:21.440 INFO 2458 --- [host-check-clockdiff2,feed99a36f42730f,8fc22d096b8e0dc7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:13:21.440 INFO 2458 --- [host-check-clockdiff3,feed99a36f42730f,0936a2698991c7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:13:21.440 INFO 2458 --- [host-check-clockdiff4,feed99a36f42730f,5613586f9638135a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:13:21.440 INFO 2458 --- [host-check-clockdiff5,feed99a36f42730f,f22df4714fd448a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:13:21.442 INFO 2458 --- [ocp-schedules-14,5d4a944c5fa18d4c,90538e9a3cc92829] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:13:21.454 INFO 2458 --- [host-check-clockdiff4,feed99a36f42730f,5613586f9638135a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534801 0 0 , error= 2025-10-27 11:13:21.471 INFO 2458 --- [host-check-clockdiff3,feed99a36f42730f,0936a2698991c7c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534801 -1 -1 , error= 2025-10-27 11:13:21.471 INFO 2458 --- [host-check-clockdiff5,feed99a36f42730f,f22df4714fd448a7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534801 0 0 , error= 2025-10-27 11:13:21.477 INFO 2458 --- [host-check-clockdiff1,feed99a36f42730f,85f9822ac9a75efd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534801 0 0 , error= 2025-10-27 11:13:21.505 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:13:21.510 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:13:21.510 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:13:21.510 INFO 2458 --- [host-check-agent-status1,1531e329982b01cb,09fc226e47a60442] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:13:21.518 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:13:21.520 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:13:21.521 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:13:21.521072068+08:00 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:13:21.521+08:00 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-agent-status2,1531e329982b01cb,4706b9c3d45a8d71] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:13:21.522 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:13:21.523 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:13:21.522083084+08:00 2025-10-27 11:13:21.524 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:13:21.524 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:13:21.525 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:13:21.525 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:13:21.525 INFO 2458 --- [host-check-agent-status3,1531e329982b01cb,f3b578164c80338e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:13:21.526 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:13:21.525090963+08:00 2025-10-27 11:13:21.526 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:13:21.523+08:00 2025-10-27 11:13:21.526 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:13:21.526+08:00 2025-10-27 11:13:21.527 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:13:21.527 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:13:21.527 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:13:21.527 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:13:21.527 INFO 2458 --- [host-check-agent-status4,1531e329982b01cb,e4dfb21da706bc5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:13:21.528 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:13:21.527731165+08:00 2025-10-27 11:13:21.529 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:13:21.528+08:00 2025-10-27 11:13:21.530 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:13:21.521+08:00 and status AVAILABLE 2025-10-27 11:13:21.530 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:13:21.530 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:13:21.530 INFO 2458 --- [host-check-agent-status5,1531e329982b01cb,b70032b94a9dea09] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:13:21.534 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:13:21.526+08:00 and status AVAILABLE 2025-10-27 11:13:21.534 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:13:21.523+08:00 and status AVAILABLE 2025-10-27 11:13:21.534 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:13:21.534 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:13:21.535 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:13:21.535414196+08:00 2025-10-27 11:13:21.535 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:13:21.535+08:00 2025-10-27 11:13:21.536 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:13:21.528+08:00 and status AVAILABLE 2025-10-27 11:13:21.538 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:13:21.540 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:13:21.540 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:13:21.544 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:13:21.535+08:00 and status AVAILABLE 2025-10-27 11:13:21.545 INFO 2458 --- [ocp-schedules-26,52a8837830911a5f,4920fd5eb1a014fa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:13:21.548 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:13:21.549 INFO 2458 --- [ocp-schedules-3,ef12c6024398eaae,b26b08e92918a262] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:13:21.549 INFO 2458 --- [ocp-schedules-26,52a8837830911a5f,4920fd5eb1a014fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:21.554 INFO 2458 --- [ocp-schedules-3,ef12c6024398eaae,b26b08e92918a262] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:21.554 INFO 2458 --- [ocp-schedules-7,23de764fab281794,933b47557ac68e11] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:13:21.554 INFO 2458 --- [ocp-schedules-3,ef12c6024398eaae,b26b08e92918a262] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:13:21.559 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:21.559 INFO 2458 --- [ocp-schedules-7,23de764fab281794,933b47557ac68e11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:21.563 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:21.564 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:21.564 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:21.564 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:21.568 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:21.569 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:21.569 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:21.571 INFO 2458 --- [ocp-schedules-3,4fc78730589e830c,f57b93a2af939ef6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:13:21.574 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:21.577 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:21.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:21.598 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:13:21.637 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.637 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.638 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.638 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:21.642 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:21.643 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:21.649 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:13:21.649 INFO 2458 --- [ocp-schedules-26,52a8837830911a5f,4920fd5eb1a014fa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:13:21.654 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:21.655 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.656 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:21.658 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:13:21.670 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:21.672 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:13:21.677 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:13:21.678 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:21.690 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:13:21.697 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:13:21.711 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:13:21.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:21.716903+08:00, lastProcessTime=2025-10-27T11:13:11.713032+08:00 2025-10-27 11:13:21.728 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:13:21.729 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.729 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.730 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.730 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.730 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.731 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:21.736 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:13:21.741 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:21.741 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.742 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:13:21.748 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:13:21.748 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:21.752 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:13:21.764 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:13:21.769 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:21.775 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:21.788 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:21.800 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:21.801 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:13:21.804 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:13:21.805 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:13:21.808 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:13:21.822 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:13:21.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:21.830 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.833 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:13:21.834 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:21.843 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:21.846 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:13:21.847 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:13:21.848 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.852 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.853 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:13:21.855 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:13:21.857 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:21.857 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:21.858 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:13:21.879 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:21.884 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:21.885 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:21.885 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:21.885 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:21.895 INFO 2458 --- [host-check-host-status4,44687dcd8417d64e,fa958864e4a16641] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:13:21.901 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:13:21.903 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:13:21.913 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:13:21.917 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:13:21.937 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:21.938 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.938 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:21.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:13:21.943 INFO 2458 --- [host-check-host-status3,44687dcd8417d64e,2d54ade00a8813d7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:13:21.943 INFO 2458 --- [host-check-host-status5,44687dcd8417d64e,9297bce5bc4a08a3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:13:21.952 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:13:21.956 INFO 2458 --- [host-check-host-status2,44687dcd8417d64e,5cfa308d9e714a17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:13:21.961 INFO 2458 --- [host-check-host-status1,44687dcd8417d64e,baf15a80fcdf2ab2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:13:21.968 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:21.970 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.977 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.978 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:21.979 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:21.979 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:21.980 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:21.982 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:13:21.984 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:21.999 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:21.716903+08:00 2025-10-27 11:13:22.033 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.033 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:22.035 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:13:22.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:22.079 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:13:22.080 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.080 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:13:22.082 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.083 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:13:22.102 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:13:22.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:22.137 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:22.138 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.139 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:13:22.169 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:13:22.169 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:13:22.193 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.193 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:22.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.321 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:22.326 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:13:22.338 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:13:22.348 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:22.401 INFO 2458 --- [host-check-clockdiff2,feed99a36f42730f,8fc22d096b8e0dc7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534802 -1 -1 , error= 2025-10-27 11:13:22.418 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:13:22.477 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:13:22.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.478 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.479 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:13:22.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.481 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.482 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:13:22.517 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:13:22.528 WARN 2458 --- [ocp-async-8,f9a52b045a65b2fe,6e92a771382775c0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:22.528 INFO 2458 --- [ocp-async-10,1308e58d23e3653f,47fe17131b79b754] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:22.560 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:22.560 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.560 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.560 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:22.560 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:22.561 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:22.562 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:13:22.604 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:13:22.604 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:13:22.604 INFO 2458 --- [ocp-schedules-7,23de764fab281794,933b47557ac68e11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:13:23.707 INFO 2458 --- [prometheus-minute-schedule-1,e022040994ba5783,b1fb82534284b25d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:23.716 INFO 2458 --- [prometheus-minute-schedule-1,e022040994ba5783,b1fb82534284b25d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:23.746 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=1000, affectRows=1000 2025-10-27 11:13:23.749 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=996, validCount=985 2025-10-27 11:13:24.006 INFO 2458 --- [ocp-schedules-26,fed23d8fc4b7b26a,4bb217ebcc077f85] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:24.009 INFO 2458 --- [ocp-schedules-26,fed23d8fc4b7b26a,4bb217ebcc077f85] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:24.107 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:24.108 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:24.356 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534720, nodeCount=1000, affectRows=1000 2025-10-27 11:13:24.360 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534720, nodeCount=1000, validCount=994 2025-10-27 11:13:24.528 WARN 2458 --- [ocp-async-10,6e9a16e0b1cc0f14,aa8db6104c54662a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:24.528 INFO 2458 --- [ocp-async-6,d1e40b30a23190f2,d02fa9cc99a97fc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:25.536 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,172b13a905418412,61aa79075dcc4d5b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=172b13a905418412, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:13:25.536 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,172b13a905418412,61aa79075dcc4d5b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=172b13a905418412, duration=1 ms] 2025-10-27 11:13:26.116 INFO 2458 --- [ocp-schedules-7,c3bbd01d63f192b7,d4aaf397acfae6a0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:26.528 WARN 2458 --- [ocp-async-6,c61681d415001bdd,b4b5782f910389b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:26.529 INFO 2458 --- [ocp-async-6,d5f12cdc2b099d43,4cff3d1add1e5503] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:26.707 INFO 2458 --- [prometheus-minute-schedule-1,a5dae0aa62d90c29,8f4056057bbb1af4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:26.714 INFO 2458 --- [prometheus-minute-schedule-1,a5dae0aa62d90c29,8f4056057bbb1af4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:26.927 INFO 2458 --- [ocp-schedules-15,b1d7985d0e418028,9cceb116a8f72a97] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:13:26.935 INFO 2458 --- [ocp-schedules-15,b1d7985d0e418028,9cceb116a8f72a97] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:26.935 INFO 2458 --- [ocp-schedules-15,b1d7985d0e418028,9cceb116a8f72a97] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:27.108 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:27.109 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:28.515 INFO 2458 --- [ocp-async-6,aaf49724a4cb31fd,dd5a1f3c7a401ebe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:28.529 WARN 2458 --- [ocp-async-8,dfc34b1c8568ca7c,1a0ffbf53db5f61d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:28.529 INFO 2458 --- [ocp-async-8,9c338f87a5fafbc9,459791051c38d682] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:28.650 INFO 2458 --- [ocp-schedules-13,c565fcb8c2025739,b7ca9a1a5a180507] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:28.680 WARN 2458 --- [scanner-112151780-1,c565fcb8c2025739,f50a926326b9c395] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:28.799 INFO 2458 --- [alarm-detect-1,c565fcb8c2025739,829e7bd3bc97bb34] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534780 ~ 1761534798 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534795, value=81.179)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534795, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534795=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:13:28.893 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:13:28.893 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:13:28.894 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:13:28.894 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:13:28.897 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:13:28.897 INFO 2458 --- [ocp-schedules-11,5863602fb20c2823,e4d5277a31f7fe3c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:13:29.357 INFO 2458 --- [ocp-schedules-6,fa8c8f33a6604965,cd249d1bf57079c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:13:29.361 INFO 2458 --- [ocp-schedules-6,fa8c8f33a6604965,cd249d1bf57079c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:13:29.361 INFO 2458 --- [ocp-schedules-6,fa8c8f33a6604965,cd249d1bf57079c5] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:13:29.410 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,8022e34a1f80bd6e,4b8234226deba7c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/binlogService?id=2&tenantId=6, client=10.40.198.13, traceId=8022e34a1f80bd6e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.getTenantBinlogService(Long,Long), args=2,6,] 2025-10-27 11:13:29.410 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/summary?id=2&tenantId=6, client=10.40.198.13, traceId=5ef0102e0a8108a0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantController.getTenantSummary(Long,Long), args=2,6,] 2025-10-27 11:13:29.411 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,8022e34a1f80bd6e,4b8234226deba7c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/binlogService client=10.40.198.13, traceId=8022e34a1f80bd6e, duration=12 ms] 2025-10-27 11:13:29.423 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=c9e80df5601bd902, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:13:29.425 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants?id=2&tenantId=6, client=10.40.198.13, traceId=3e4303bad4d2e499, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantStandbyController.listRelationTenants(Long,Long), args=2,6,] 2025-10-27 11:13:29.435 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2edddfd522769d0a,42d9f53180f4803c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=10.40.198.13, traceId=2edddfd522769d0a, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2025-10-27 11:13:29.458 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.459 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.463 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:13:29.469 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.470 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.474 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,2edddfd522769d0a,42d9f53180f4803c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=10.40.198.13, traceId=2edddfd522769d0a, duration=60 ms] 2025-10-27 11:13:29.476 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:13:29.476 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.476 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.481 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.482 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:13:29.485 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:13:29.490 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.490 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.493 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.493 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.496 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:13:29.501 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.504 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:13:29.509 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.510 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.510 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.511 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:13:29.518 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.527 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:13:29.538 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.543 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:13:29.546 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.547 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [_lcl_op_interval] 2025-10-27 11:13:29.549 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.553 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:13:29.554 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.557 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:13:29.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a0f04e52d606ec7f,0c09bb343a06fd8d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/unitSpecs/limitRule?clusterId=2, client=10.40.198.13, traceId=a0f04e52d606ec7f, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObUnitSpecController.getUnitSpecLimitRule(Long), args=2,] 2025-10-27 11:13:29.574 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.579 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1] 2025-10-27 11:13:29.582 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.582 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.604 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.605 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [resource_hard_limit] 2025-10-27 11:13:29.630 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.635 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.706 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,c9e80df5601bd902,cd457265c079da78] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=c9e80df5601bd902, duration=292 ms] 2025-10-27 11:13:29.709 INFO 2458 --- [prometheus-minute-schedule-1,f646f7e6cdd9b5c7,d2d6250438617a21] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:29.716 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.717 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.721 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.727 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.728 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:13:29.728 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.729 INFO 2458 --- [prometheus-minute-schedule-1,f646f7e6cdd9b5c7,d2d6250438617a21] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:29.729 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:29.739 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.739 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.740 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:13:29.772 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.774 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:13:29.774 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.777 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:13:29.794 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a0f04e52d606ec7f,0c09bb343a06fd8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.795 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a0f04e52d606ec7f,0c09bb343a06fd8d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ ZONE, SVR_IP, SVR_PORT, NAME, VALUE, DATA_TYPE, SECTION, EDIT_LEVEL, INFO FROM GV$OB_PARAMETERS WHERE SCOPE = 'CLUSTER' AND name = ?, args: [__min_full_resource_pool_memory] 2025-10-27 11:13:29.801 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a0f04e52d606ec7f,0c09bb343a06fd8d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/unitSpecs/limitRule client=10.40.198.13, traceId=a0f04e52d606ec7f, duration=256 ms] 2025-10-27 11:13:29.802 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.802 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.806 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.806 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [ob_enable_sql_audit] 2025-10-27 11:13:29.806 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1] 2025-10-27 11:13:29.810 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.825 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.825 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.825 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:29.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:13:29.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:13:29.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:13:29.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5ef0102e0a8108a0,71609b3913eae80f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/summary client=10.40.198.13, traceId=5ef0102e0a8108a0, duration=431 ms] 2025-10-27 11:13:29.833 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:13:29.833 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.o.i.t.c.TenantOperatorHelperImpl : These operators for clusterIds=[] have been created. 2025-10-27 11:13:29.833 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3e4303bad4d2e499,c998c0dce8f0d4aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants client=10.40.198.13, traceId=3e4303bad4d2e499, duration=419 ms] 2025-10-27 11:13:29.954 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=e0ddb1a83e8775ca, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:13:29.955 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,7bb456400563df9b,9eebebfe98e9bd02] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/zones?id=2&tenantId=6, client=10.40.198.13, traceId=7bb456400563df9b, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listObClusterZones(Long,Long), args=2,6,] 2025-10-27 11:13:29.955 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c35e221eaf31e6b6,605df1df3cb74636] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/binlogService?id=2&tenantId=6, client=10.40.198.13, traceId=c35e221eaf31e6b6, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.getTenantBinlogService(Long,Long), args=2,6,] 2025-10-27 11:13:29.955 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants?id=2&tenantId=6, client=10.40.198.13, traceId=42ed48ec439bc1e6, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantStandbyController.listRelationTenants(Long,Long), args=2,6,] 2025-10-27 11:13:29.957 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,c35e221eaf31e6b6,605df1df3cb74636] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/binlogService client=10.40.198.13, traceId=c35e221eaf31e6b6, duration=11 ms] 2025-10-27 11:13:29.998 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,7bb456400563df9b,9eebebfe98e9bd02] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/zones client=10.40.198.13, traceId=7bb456400563df9b, duration=51 ms] 2025-10-27 11:13:29.998 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:29.999 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.003 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:13:30.011 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:30.011 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.017 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:13:30.022 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.025 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') AND tenant_id = ?, args: [1002] 2025-10-27 11:13:30.032 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.032 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:13:30.033 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.033 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:30.052 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.054 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:13:30.055 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.055 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:13:30.057 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.057 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:13:30.095 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e0ddb1a83e8775ca,0474867d6b93e0ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=e0ddb1a83e8775ca, duration=149 ms] 2025-10-27 11:13:30.095 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.096 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:30.109 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:30.113 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:30.162 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.162 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:13:30.166 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:30.167 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.168 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `collation_name` AS collation, `character_set_name` AS charset, `id`, `is_default` FROM information_schema.collations 2025-10-27 11:13:30.189 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.191 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ incarnation, round_id AS log_archive_round, tenant_id, path AS backup_dest, start_scn_display AS min_first_time, checkpoint_scn_display AS max_next_time, status, truncate((time_to_usec(now()) - checkpoint_scn / 1000) / 1000000, 4) AS delay, now(6) AS check_time FROM CDB_OB_ARCHIVELOG_SUMMARY WHERE tenant_id = ? AND status = 'DOING', args: [1002] 2025-10-27 11:13:30.210 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:30.211 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:30.212 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, svr_port, tenant_id, zone, resource_name, current_utilization, max_utilization, reserved_value, limit_value, effective_limit_type from GV$OB_TENANT_RESOURCE_LIMIT where tenant_id = ?, args: [1002] 2025-10-27 11:13:30.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.b.i.BinlogClusterMetadbHelperImpl : These binlog clusters:=[] use tenant=6 as metadb. 2025-10-27 11:13:30.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.o.i.t.c.TenantOperatorHelperImpl : These operators for clusterIds=[] have been created. 2025-10-27 11:13:30.217 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,42ed48ec439bc1e6,8e15dfdb14353157] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/6/listRelationTenants client=10.40.198.13, traceId=42ed48ec439bc1e6, duration=271 ms] 2025-10-27 11:13:30.529 WARN 2458 --- [ocp-async-8,a0df637a9190095c,c1494a1b67ebe64f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:30.529 INFO 2458 --- [ocp-async-8,edccc7ca57c853eb,8c1d9de16ba85648] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:31.716185+08:00, lastProcessTime=2025-10-27T11:13:21.716903+08:00 2025-10-27 11:13:31.968 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:31.716185+08:00 2025-10-27 11:13:32.529 WARN 2458 --- [ocp-async-7,f758e901e52d86a0,2e5642a62929d5fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:32.529 INFO 2458 --- [ocp-async-8,630467fc53e42d94,6f282b031e40bbf3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:32.707 INFO 2458 --- [prometheus-minute-schedule-1,27468b4d880df74a,240292fdd154eb29] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:32.715 INFO 2458 --- [prometheus-minute-schedule-1,27468b4d880df74a,240292fdd154eb29] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:33.113 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:33.114 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:33.787 INFO 2458 --- [ocp-schedules-13,9c0dc5a359494afa,2ea67b33326d7e4f] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:13:33.787 INFO 2458 --- [ocp-schedules-13,9c0dc5a359494afa,2ea67b33326d7e4f] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:13:34.018 INFO 2458 --- [ocp-schedules-24,99270493ece0486c,1f72a2a142a30cd3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:34.021 INFO 2458 --- [ocp-schedules-24,99270493ece0486c,1f72a2a142a30cd3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:34.529 WARN 2458 --- [ocp-async-8,89eebf5d281bde01,dab9616a4b362793] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:34.529 INFO 2458 --- [ocp-async-10,75f39486e42c3dc9,72f83d423da57fcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:35.707 INFO 2458 --- [prometheus-minute-schedule-1,5c5f331bd9c67f44,5016681113cfbfb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:35.719 INFO 2458 --- [prometheus-minute-schedule-1,5c5f331bd9c67f44,5016681113cfbfb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:36.114 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:36.115 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:36.123 INFO 2458 --- [ocp-schedules-28,75d05b76cf3d8a7a,82853b58f74a2581] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:36.205 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:13:36.205 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:13:36.277 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:36.277 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:36.332 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:36.332 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:36.332 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:36.333 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:13:36.347 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:36.347 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:36.348 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:36.348 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:13:36.420 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:36.422 INFO 2458 --- [ocp-schedules-4,0914d7fe827b6e9b,4140d92c88ecce91] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:13:36.529 WARN 2458 --- [ocp-async-3,0528440206669555,bc6a4020ae6f8206] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:36.529 INFO 2458 --- [ocp-async-10,2f8fe62ca1ab3043,2207c7c4e1a8bc69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:37.708 INFO 2458 --- [prometheus-cache-monitor1,02c1e30d7d9744f1,aa2a0bd88783582c] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20307,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5078,"seriesIdKeySize":8771,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:13:37.708 INFO 2458 --- [prometheus-cache-monitor1,02c1e30d7d9744f1,aa2a0bd88783582c] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:13:37.821 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:37.822 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:13:37.822 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:13:37.822 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:13:37.822 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:13:37.822 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:13:37.834 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:37.834 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:37.834 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:37.835 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:37.836 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:37.838 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:37.838 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:37.839 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:37.839 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:37.840 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:37.889 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:13:37.890 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:13:37.892 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:13:37.892 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:13:37.893 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:13:37.894 INFO 2458 --- [ocp-schedules-17,71f61f03958cf101,714e56c55b9186ae] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:13:38.515 INFO 2458 --- [ocp-async-10,37ef1e60bdf25f27,4739b398b803a959] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:38.529 WARN 2458 --- [ocp-async-7,ce4618811039a854,f96af5d7bd12e672] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:38.529 INFO 2458 --- [ocp-async-9,07be12e52618a3f5,617fef3dec25b4a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:38.634 INFO 2458 --- [ocp-schedules-8,0f203b031f32d2df,21437c7c89177d1f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:13:38.640 INFO 2458 --- [ocp-schedules-8,0f203b031f32d2df,21437c7c89177d1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:38.640 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:13:38.642 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:13:38.643 INFO 2458 --- [ocp-schedules-8,0f203b031f32d2df,21437c7c89177d1f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:13:38.707 INFO 2458 --- [prometheus-minute-schedule-1,bdbb46dc3de5f93a,aa25913e2bc468e6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:38.714 INFO 2458 --- [prometheus-minute-schedule-1,bdbb46dc3de5f93a,aa25913e2bc468e6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:38.804 INFO 2458 --- [ocp-schedules-21,40affe17d6c0dbd0,c405c5eb409303a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:38.827 WARN 2458 --- [scanner-112152041-1,40affe17d6c0dbd0,ee24a41f4be917a9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:39.115 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:39.117 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:40.013 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:13:40.015 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:13:40.049 INFO 2458 --- [Check obproxy1,25b4c29a34ecd886,b1ae36fb12b6587b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:13:40.049 INFO 2458 --- [Check obproxy1,25b4c29a34ecd886,b1ae36fb12b6587b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:13:40.053 INFO 2458 --- [Collect obproxy1,859276126c20ae2e,1ce4d32d38d523e7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:13:40.053 INFO 2458 --- [Collect obproxy1,859276126c20ae2e,1ce4d32d38d523e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:13:40.080 INFO 2458 --- [Check obproxy1,25b4c29a34ecd886,b1ae36fb12b6587b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:13:40.082 INFO 2458 --- [Check obproxy1,25b4c29a34ecd886,b1ae36fb12b6587b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:13:40.082 INFO 2458 --- [Check obproxy1,25b4c29a34ecd886,b1ae36fb12b6587b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:13:40.083 INFO 2458 --- [Collect obproxy1,859276126c20ae2e,1ce4d32d38d523e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:13:40.086 INFO 2458 --- [Collect obproxy1,859276126c20ae2e,1ce4d32d38d523e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:13:40.088 INFO 2458 --- [Collect obproxy1,859276126c20ae2e,1ce4d32d38d523e7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:13:40.530 INFO 2458 --- [ocp-async-8,06e9c70a9d3ddfaa,5b040173b0210b9e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:40.530 WARN 2458 --- [ocp-async-6,5b0090c460f11172,f747fd04417c141f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:40.885 INFO 2458 --- [ocp-schedules-6,52984d74ddb549ee,f8bd202646e9237e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:13:40.886 INFO 2458 --- [ocp-schedules-6,52984d74ddb549ee,f8bd202646e9237e] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:13:41.707 INFO 2458 --- [prometheus-minute-schedule-1,5061df78a747d57b,b24abafdfb17a922] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:41.715 INFO 2458 --- [prometheus-minute-schedule-1,5061df78a747d57b,b24abafdfb17a922] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:41.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:41.716482+08:00, lastProcessTime=2025-10-27T11:13:31.716185+08:00 2025-10-27 11:13:41.940 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:41.716482+08:00 2025-10-27 11:13:42.117 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:42.118 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:42.498 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,cf89a330d08a8867,688baa2656cabf7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=cf89a330d08a8867, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:13:42.499 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,cf89a330d08a8867,688baa2656cabf7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=cf89a330d08a8867, duration=1 ms] 2025-10-27 11:13:42.530 WARN 2458 --- [ocp-async-3,f07f8c143a0013c1,ccdd000cc5ba427b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:42.530 INFO 2458 --- [ocp-async-1,de0c501244715445,ed5750216da225dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:44.029 INFO 2458 --- [ocp-schedules-10,606e60b34485482b,fdd9d1e462d03496] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:44.033 INFO 2458 --- [ocp-schedules-10,606e60b34485482b,fdd9d1e462d03496] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:44.045 INFO 2458 --- [ocp-schedules-5,c40599253bec2711,254623f99bb759af] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:13:44.530 WARN 2458 --- [ocp-async-5,e89da4ac6ca4bc0a,c7f9afc131ed7fa2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:44.530 INFO 2458 --- [ocp-async-10,efb0ea3358adad61,4093bcc3d79c082f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:44.707 INFO 2458 --- [prometheus-minute-schedule-1,3b67a4d34f27eca6,11c3a2966174795f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:44.715 INFO 2458 --- [prometheus-minute-schedule-1,3b67a4d34f27eca6,11c3a2966174795f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:45.102 INFO 2458 --- [prometheus-second-schedule-1,f4ce18f72aa11473,ab8f9e423cdcdc57] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:13:45.119 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:45.122 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:46.129 INFO 2458 --- [ocp-schedules-20,8097379a0b735e96,89fac6c06ba7ed6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:46.530 INFO 2458 --- [ocp-async-6,7432659b7671f943,56f178216249179c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:46.530 WARN 2458 --- [ocp-async-2,98a646ae9a5ed01c,6780676c763b7dc3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:46.939 INFO 2458 --- [ocp-schedules-13,8b0de7bafb81545d,7d8f053c6575ba26] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:13:46.946 INFO 2458 --- [ocp-schedules-13,8b0de7bafb81545d,7d8f053c6575ba26] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:46.946 INFO 2458 --- [ocp-schedules-13,8b0de7bafb81545d,7d8f053c6575ba26] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:13:47.120 INFO 2458 --- [ocp-schedules-30,5dc5527849f7ea1d,e5f4b7e039a212a5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:13:47.120 INFO 2458 --- [ocp-schedules-30,5dc5527849f7ea1d,e5f4b7e039a212a5] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:13:47.128 INFO 2458 --- [ocp-schedules-30,5dc5527849f7ea1d,e5f4b7e039a212a5] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:13:47.128 INFO 2458 --- [ocp-schedules-30,5dc5527849f7ea1d,e5f4b7e039a212a5] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:13:47.707 INFO 2458 --- [prometheus-minute-schedule-1,e146b735c8a2997f,595942376a5c54aa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:47.716 INFO 2458 --- [prometheus-minute-schedule-1,e146b735c8a2997f,595942376a5c54aa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:48.122 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:48.123 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:48.218 INFO 2458 --- [ocp-schedules-19,61307f38468ca637,364be766567aca17] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:13:48.218+08:00, clusters=[1, 2] 2025-10-27 11:13:48.232 INFO 2458 --- [ocp-schedules-19,61307f38468ca637,364be766567aca17] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:13:48.350 INFO 2458 --- [ocp-schedules-19,61307f38468ca637,364be766567aca17] c.o.o.p.s.i.alarm.SlowSqlDataSource : collect SlowSqlData for myoceanbase2, save 1 to SqlAggregateData 2025-10-27 11:13:48.515 INFO 2458 --- [ocp-async-8,12da05098c1ea59e,85c52c3de06887cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:48.530 WARN 2458 --- [ocp-async-1,1146674b7174c31e,b4a591c740757243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:48.530 INFO 2458 --- [ocp-async-3,dbda09be9caefc88,e13d4b8cdda5e4c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:48.918 INFO 2458 --- [ocp-schedules-26,d83be8bf1cc64b4b,2f983f4251c72b09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:48.970 WARN 2458 --- [scanner-112152262-1,d83be8bf1cc64b4b,176fa74c565267bb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:48.993 INFO 2458 --- [alarm-detect-12,d83be8bf1cc64b4b,9a44a1b7ea6e66d7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534758 ~ 1761534818 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=203114.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203114.56, values={1761534780=203114.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:13:48.994 INFO 2458 --- [alarm-detect-12,d83be8bf1cc64b4b,9a44a1b7ea6e66d7] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:13:49.006 INFO 2458 --- [alarm-detect-15,d83be8bf1cc64b4b,122b078c5f39edce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 229, data is empty. 2025-10-27 11:13:49.008 INFO 2458 --- [alarm-detect-10,d83be8bf1cc64b4b,a3a76c920cf25975] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 229, data is empty. 2025-10-27 11:13:49.008 INFO 2458 --- [alarm-detect-1,d83be8bf1cc64b4b,eb5f3489c032f794] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 229, data is empty. 2025-10-27 11:13:49.011 INFO 2458 --- [alarm-detect-15,d83be8bf1cc64b4b,122b078c5f39edce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:13:49.013 INFO 2458 --- [alarm-detect-10,d83be8bf1cc64b4b,a3a76c920cf25975] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:13:49.014 INFO 2458 --- [alarm-detect-1,d83be8bf1cc64b4b,eb5f3489c032f794] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:13:49.046 INFO 2458 --- [alarm-detect-8,d83be8bf1cc64b4b,a732ca7257207710] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534759 ~ 1761534819 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=93.027)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.027, values={1761534780=93.027}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:13:49.047 INFO 2458 --- [alarm-detect-8,d83be8bf1cc64b4b,a732ca7257207710] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:13:49.063 INFO 2458 --- [alarm-detect-2,d83be8bf1cc64b4b,5f8872f1ee392bd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 229, data is empty. 2025-10-27 11:13:49.067 INFO 2458 --- [alarm-detect-2,d83be8bf1cc64b4b,5f8872f1ee392bd4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:13:49.067 INFO 2458 --- [alarm-detect-5,d83be8bf1cc64b4b,60acff84d5579c95] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534759 ~ 1761534819 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=203115.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203115.912, values={1761534780=203115.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:13:49.068 INFO 2458 --- [alarm-detect-5,d83be8bf1cc64b4b,60acff84d5579c95] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:13:49.096 INFO 2458 --- [alarm-detect-2,d83be8bf1cc64b4b,397c5238e5f9a6d4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534795 ~ 1761534819 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534810, value=82.671)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534810, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534810=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:13:50.530 WARN 2458 --- [ocp-async-7,cfdf31a3c065addd,5e8d09099b3473a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:50.530 INFO 2458 --- [ocp-async-7,961fc339385d2a23,846f40649b53dcc4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:50.707 INFO 2458 --- [prometheus-minute-schedule-1,d1ecca39c880e87e,9c82fb39a6a14c5e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:50.715 INFO 2458 --- [prometheus-minute-schedule-1,d1ecca39c880e87e,9c82fb39a6a14c5e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:51.123 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:51.123 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:51.124 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,01fe7ffa20c906ab,4132459fa2b798b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=01fe7ffa20c906ab, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:13:51.134 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,01fe7ffa20c906ab,4132459fa2b798b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=01fe7ffa20c906ab, duration=16 ms] 2025-10-27 11:13:51.135 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,b3025985abe55fb9,de4f3f7b89effb3d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=b3025985abe55fb9, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:13:51.135 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,b3025985abe55fb9,de4f3f7b89effb3d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=b3025985abe55fb9, duration=6 ms] 2025-10-27 11:13:51.146 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,b6287dcf16c3502e,220c6d851f6c79e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=b6287dcf16c3502e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:13:51.147 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,b6287dcf16c3502e,220c6d851f6c79e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=b6287dcf16c3502e, duration=8 ms] 2025-10-27 11:13:51.153 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,fee10ed93c416465,bfdc3bdc82f0c449] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=fee10ed93c416465, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:13:51.272 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,2a92248abe25b6a1,daea933dbdbea1ce] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=2a92248abe25b6a1, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:13:51.322 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,2a92248abe25b6a1,daea933dbdbea1ce] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=2a92248abe25b6a1, duration=58 ms] 2025-10-27 11:13:51.323 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:13:51.323 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:13:51.323 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:13:51.365 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,bb8901555bb6273f,71615aaadacbf10e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=CLUSTER&type=TOP, client=10.40.198.13, traceId=bb8901555bb6273f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,CLUSTER,OBCLUSTER,] 2025-10-27 11:13:51.366 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fb1c5feeee184a11,ad036d38829a17b5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration?limit=5&inDays=3, client=10.40.198.13, traceId=fb1c5feeee184a11, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getClustersTopAvgCompactionDuration(Integer,Integer), args=3,5,] 2025-10-27 11:13:51.368 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1abb32db7a6b8348,0f0a5dc0c5e6f761] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id&metrics=sql_all_count&limit=5&startTime=2025-10-27T10%3A13%3A51%2B08%3A00&endTime=2025-10-27T11%3A13%3A51%2B08%3A00&maxPoints=360, client=10.40.198.13, traceId=1abb32db7a6b8348, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:13:51+08:00,2025-10-27T11:13:51+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,0,0,5,360,app:OB,true,] 2025-10-27 11:13:51.368 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,2e9c9050f650edb4,89ed89aefed8e63a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/alarmSummaries?null, client=10.40.198.13, traceId=2e9c9050f650edb4, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listClusterAlarmSummaries(String,List), args=] 2025-10-27 11:13:51.376 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,8d309244f602ae98,890ba3411a9d7408] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration?limit=5&inDays=7, client=10.40.198.13, traceId=8d309244f602ae98, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantCompactionController.topAvgCompactionDuration(Integer,Integer), args=7,5,] 2025-10-27 11:13:51.390 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,bb8901555bb6273f,71615aaadacbf10e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=bb8901555bb6273f, duration=31 ms] 2025-10-27 11:13:51.398 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fb1c5feeee184a11,ad036d38829a17b5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/stats/topAvgCompactionDuration client=10.40.198.13, traceId=fb1c5feeee184a11, duration=39 ms] 2025-10-27 11:13:51.402 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,1abb32db7a6b8348,0f0a5dc0c5e6f761] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=1abb32db7a6b8348, duration=43 ms] 2025-10-27 11:13:51.417 INFO 2458 --- [ocp-schedules-26,44a4d78cdd9035bf,ad52cd6b917ee38f] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:51.418 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:13:51.418 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:13:51.418 INFO 2458 --- [ocp-schedules-9,f0bdc8d848180bbb,73fb5f78a572a10a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:51.418 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:13:51.422 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:13:51.418 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:13:51.421 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:13:51.421 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:13:51.423 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:13:51.429 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:13:51.432 INFO 2458 --- [ocp-schedules-29,1b9d76858e3a4bac,bbd7b8e50ed3095e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:13:51.435 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:51.435 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:51.436 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:51.436 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:51.441 INFO 2458 --- [host-check-clockdiff2,1b9d76858e3a4bac,f355e943899ba218] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:13:51.442 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:51.442 INFO 2458 --- [ocp-schedules-14,9ba5c4c55d5ffdd8,2cb85c4fb1aa4015] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:13:51.442 INFO 2458 --- [host-check-clockdiff1,1b9d76858e3a4bac,81fe13ad0c48d1d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:13:51.443 INFO 2458 --- [host-check-clockdiff3,1b9d76858e3a4bac,68eed53b6739dfa2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:13:51.446 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:51.449 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:51.450 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,fee10ed93c416465,bfdc3bdc82f0c449] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=fee10ed93c416465, duration=310 ms] 2025-10-27 11:13:51.451 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:51.452 INFO 2458 --- [host-check-clockdiff4,1b9d76858e3a4bac,09d7531c23bfd103] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:13:51.453 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:51.453 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:51.454 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:13:51.454 INFO 2458 --- [host-check-clockdiff5,1b9d76858e3a4bac,ea010d601ec732a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:13:51.454 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,2e9c9050f650edb4,89ed89aefed8e63a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/alarmSummaries client=10.40.198.13, traceId=2e9c9050f650edb4, duration=95 ms] 2025-10-27 11:13:51.471 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:51.472 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:51.472 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:51.474 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:51.475 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:51.475 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:51.475 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:51.476 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:51.478 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:51.478 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:51.532 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:13:51.537 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:13:51.537 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:13:51.537 INFO 2458 --- [host-check-agent-status2,44a4d78cdd9035bf,3417b26034e9cfac] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:13:51.545 INFO 2458 --- [ocp-schedules-11,ef310005ed1be3b8,0da7f87e0b75528d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:13:51.547 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,8d309244f602ae98,890ba3411a9d7408] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/tenants/topAvgCompactionDuration client=10.40.198.13, traceId=8d309244f602ae98, duration=181 ms] 2025-10-27 11:13:51.548 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:13:51.549 INFO 2458 --- [ocp-schedules-30,3ae69a50424e1643,9bbeeacad32ce300] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:13:51.550 INFO 2458 --- [ocp-schedules-11,ef310005ed1be3b8,0da7f87e0b75528d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:51.554 INFO 2458 --- [ocp-schedules-1,856c5811606bea03,5db747fad571fb66] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:13:51.554 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:13:51.554 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:13:51.554 INFO 2458 --- [host-check-agent-status4,44a4d78cdd9035bf,31353ea7a02fbd88] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:13:51.555 INFO 2458 --- [ocp-schedules-30,3ae69a50424e1643,9bbeeacad32ce300] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:51.556 INFO 2458 --- [host-check-clockdiff3,1b9d76858e3a4bac,68eed53b6739dfa2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534831 -1 -1 , error= 2025-10-27 11:13:51.556 INFO 2458 --- [ocp-schedules-30,3ae69a50424e1643,9bbeeacad32ce300] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:13:51.556 INFO 2458 --- [host-check-clockdiff4,1b9d76858e3a4bac,09d7531c23bfd103] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534831 -1 -1 , error= 2025-10-27 11:13:51.556 INFO 2458 --- [host-check-clockdiff5,1b9d76858e3a4bac,ea010d601ec732a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534831 0 0 , error= 2025-10-27 11:13:51.574 INFO 2458 --- [ocp-schedules-1,856c5811606bea03,5db747fad571fb66] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:51.575 INFO 2458 --- [ocp-schedules-28,d5c9f33357d17d1c,e36122345aa30113] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:13:51.575 INFO 2458 --- [ocp-schedules-5,fb9da4f439dfb36a,c146f21cbec8e8a0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:13:51.586 INFO 2458 --- [ocp-schedules-24,c97c5468e7d2843e,3326b827c8f21712] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:13:51.587 INFO 2458 --- [ocp-schedules-28,d5c9f33357d17d1c,e36122345aa30113] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:13:51.588 INFO 2458 --- [host-check-clockdiff1,1b9d76858e3a4bac,81fe13ad0c48d1d2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534831 0 0 , error= 2025-10-27 11:13:51.591 INFO 2458 --- [host-check-clockdiff2,1b9d76858e3a4bac,f355e943899ba218] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534831 -1 -1 , error= 2025-10-27 11:13:51.593 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clustersView?page=1&size=10&sort=name%2Casc, client=10.40.198.13, traceId=ce53957e0038f982, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClustersView(String,List,WithRemote,List,Pageable), args=NONE,Page request [number: 0, size 10, sort: name: ASC],] 2025-10-27 11:13:51.596 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:13:51.597 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:13:51.599 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:13:51.597719479+08:00 2025-10-27 11:13:51.600 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:13:51.600 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:13:51.600 INFO 2458 --- [host-check-agent-status3,44a4d78cdd9035bf,ebd9100b18c394cd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:13:51.602 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:13:51.602 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:13:51.599+08:00 2025-10-27 11:13:51.602 INFO 2458 --- [ocp-schedules-5,fb9da4f439dfb36a,c146f21cbec8e8a0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:13:51.605 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:13:51.605 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:13:51.606 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:13:51.606 INFO 2458 --- [host-check-agent-status5,44a4d78cdd9035bf,61f33ce7d0140abd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:13:51.606 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:13:51.607 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:13:51.607281717+08:00 2025-10-27 11:13:51.608 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:13:51.607+08:00 2025-10-27 11:13:51.608 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:13:51.609 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:13:51.608749603+08:00 2025-10-27 11:13:51.610 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:13:51.609+08:00 2025-10-27 11:13:51.613 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:13:51.599+08:00 and status AVAILABLE 2025-10-27 11:13:51.616 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:13:51.616 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:13:51.616 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:13:51.619 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:13:51.618588896+08:00 2025-10-27 11:13:51.619 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:13:51.619 INFO 2458 --- [host-check-agent-status1,44a4d78cdd9035bf,cc4767a7e0010737] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:13:51.619 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:13:51.619674070+08:00 2025-10-27 11:13:51.620 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:13:51.607+08:00 and status AVAILABLE 2025-10-27 11:13:51.620 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:13:51.619+08:00 2025-10-27 11:13:51.620 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:13:51.619+08:00 2025-10-27 11:13:51.621 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:13:51.626 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:13:51.626 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:13:51.609+08:00 and status AVAILABLE 2025-10-27 11:13:51.631 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:13:51.637 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:13:51.619+08:00 and status AVAILABLE 2025-10-27 11:13:51.640 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:13:51.619+08:00 and status AVAILABLE 2025-10-27 11:13:51.641 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.642 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:51.642 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:13:51.643 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:13:51.645 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:13:51.656 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:51.660 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:51.664 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:51.664 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:13:51.667 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:13:51.667 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:51.674 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:13:51.674 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:13:51.676 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.676 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.677 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:13:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:51.679 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:13:51.684 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:13:51.684 INFO 2458 --- [ocp-schedules-11,ef310005ed1be3b8,0da7f87e0b75528d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:13:51.687 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.687 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:51.723 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.723 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.723 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.723 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.724 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.725 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:13:51.726 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:13:51.725350+08:00, lastProcessTime=2025-10-27T11:13:41.716482+08:00 2025-10-27 11:13:51.738 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.738 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.738 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.739 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:13:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.740 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.741 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:51.752 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:51.754 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.754 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.755 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.755 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:13:51.756 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:51.759 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:51.765 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:51.766 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:13:51.770 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:51.770 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.773 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:13:51.774 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:13:51.774 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:13:51.779 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:13:51.781 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:13:51.798 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:13:51.819 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:13:51.828 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:13:51.830 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:13:51.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.833 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:13:51.833 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:13:51.835 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.835 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:13:51.835 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:13:51.836 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.837 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:13:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.842 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.843 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:13:51.859 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:51.862 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:51.863 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:51.865 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,ce53957e0038f982,1131ff5f5b5db63d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clustersView client=10.40.198.13, traceId=ce53957e0038f982, duration=395 ms] 2025-10-27 11:13:51.866 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.867 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:51.868 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.869 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:51.869 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:51.871 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:13:51.872 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:51.872 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:13:51.879 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:13:51.883 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:13:51.886 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:13:51.886 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:13:51.895 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:51.895 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:51.904 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:13:51.910 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:13:51.912 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:13:51.918 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:13:51.921 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:13:51.926 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:51.945 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:51.946 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:51.948 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:51.957 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:13:51.965 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:13:51.966 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:13:51.967 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:13:51.971 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.975 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.975 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:13:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:51.980 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:51.981 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:13:51.981 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:13:51.982 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:51.986 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:13:51.989 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:13:51.994 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:13:52.004 INFO 2458 --- [host-check-host-status5,f0bdc8d848180bbb,567f7a1e2604398a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:13:52.005 INFO 2458 --- [host-check-host-status3,f0bdc8d848180bbb,831c0f50d821ba81] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:13:52.008 INFO 2458 --- [host-check-host-status2,f0bdc8d848180bbb,236df7608aa835e6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:13:52.012 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:13:52.015 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:13:52.020 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:13:52.033 INFO 2458 --- [host-check-host-status4,f0bdc8d848180bbb,796762b52b64e66b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:13:52.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:52.047 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:13:51.725350+08:00 2025-10-27 11:13:52.053 INFO 2458 --- [host-check-host-status1,f0bdc8d848180bbb,145804d2b1473cab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:13:52.082 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:13:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.084 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:52.126 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:13:52.126 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.127 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:13:52.129 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.129 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:13:52.147 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:13:52.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:13:52.176 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:52.176 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.177 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:13:52.195 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.195 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:13:52.199 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:13:52.199 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:13:52.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:13:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.278 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.278 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:13:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:13:52.295 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:13:52.305 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:13:52.308 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:13:52.361 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:13:52.418 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:13:52.418 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.418 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.419 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:13:52.421 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.421 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.421 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.422 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:13:52.463 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:13:52.510 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:13:52.511 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:13:52.530 INFO 2458 --- [ocp-async-1,f5a0b04035738c3b,c88994996134b41e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:52.531 WARN 2458 --- [ocp-async-8,8a9f4e15e1a0969e,82ad6ed58a37e31e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:52.549 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:13:52.549 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:13:52.549 INFO 2458 --- [ocp-schedules-1,856c5811606bea03,5db747fad571fb66] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:13:53.108 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,418eb1cd65f6eaed,5e129d4533608506] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=418eb1cd65f6eaed, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:13:53.108 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,418eb1cd65f6eaed,5e129d4533608506] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=418eb1cd65f6eaed, duration=9 ms] 2025-10-27 11:13:53.108 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,ea59873f6f76ac45,f0e8f27b02797b0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=ea59873f6f76ac45, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:13:53.108 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,ea59873f6f76ac45,f0e8f27b02797b0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=ea59873f6f76ac45, duration=9 ms] 2025-10-27 11:13:53.110 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,8bff6d7bfa8cf042,d3356ab3e0b05def] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=8bff6d7bfa8cf042, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:13:53.110 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,8bff6d7bfa8cf042,d3356ab3e0b05def] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=8bff6d7bfa8cf042, duration=10 ms] 2025-10-27 11:13:53.160 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:13:53.160 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:13:53.160 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:13:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:13:53.221 INFO 2458 --- [ocp-schedules-17,f100b628aedde6fe,75e4106fd435d721] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20307,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5084,"seriesIdKeySize":8772,"secondWriteQueue":0,"minuteWriteCacheSize":7863} 2025-10-27 11:13:53.221 INFO 2458 --- [ocp-schedules-17,f100b628aedde6fe,75e4106fd435d721] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:13:53.707 INFO 2458 --- [prometheus-minute-schedule-1,9305471386a1a4d5,294e47e756b911ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:53.717 INFO 2458 --- [prometheus-minute-schedule-1,9305471386a1a4d5,294e47e756b911ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:54.040 INFO 2458 --- [ocp-schedules-1,85f0a81055805b10,f2eeb30cc9f7c85e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:13:54.044 INFO 2458 --- [ocp-schedules-1,85f0a81055805b10,f2eeb30cc9f7c85e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:13:54.123 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:54.125 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:54.531 WARN 2458 --- [ocp-async-3,c98ac2b62523a791,7c8948f85042bd1c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:54.531 INFO 2458 --- [ocp-async-10,31b70155e48e83db,4feae7f2394f7293] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:54.705 INFO 2458 --- [ocp-schedules-5,20bbb20bbf92097a,66b7dee7f2a4eca5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:13:56.097 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,0371a588a9d13d13,f127e9834f46aff9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=0371a588a9d13d13, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:13:56.102 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,0371a588a9d13d13,f127e9834f46aff9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=0371a588a9d13d13, duration=14 ms] 2025-10-27 11:13:56.141 INFO 2458 --- [ocp-schedules-9,e557ca62de1c2d53,7ded4b4346a3fa9f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:13:56.531 WARN 2458 --- [ocp-async-5,53291735fdbe34e0,a5e39403d67dbdb8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:56.531 INFO 2458 --- [ocp-async-5,6df98e4741f8a2f3,61ed5dbd15517a35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:56.707 INFO 2458 --- [prometheus-minute-schedule-1,23209c424a803a5d,ed894ecc88fd7b06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:56.715 INFO 2458 --- [prometheus-minute-schedule-1,23209c424a803a5d,ed894ecc88fd7b06] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:13:57.125 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:13:57.126 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:13:57.845 INFO 2458 --- [ocp-schedules-21,43a1da8be85979fc,d067a74ce9ffc7c9] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:13:57.852 INFO 2458 --- [ocp-schedules-21,43a1da8be85979fc,d067a74ce9ffc7c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:13:57.893 INFO 2458 --- [ocp-schedules-21,43a1da8be85979fc,d067a74ce9ffc7c9] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:13:58.515 INFO 2458 --- [ocp-async-2,5053e172ec521150,f1f93aed00c833d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:13:58.531 WARN 2458 --- [ocp-async-8,3e9ec79805d99cd4,cffd2189a0b6d155] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:13:58.531 INFO 2458 --- [ocp-async-6,a90e68c538050be2,976af5370e0ff9e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:13:59.102 INFO 2458 --- [ocp-schedules-23,80bfb11a779d0060,e54592e39a974797] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:13:59.133 WARN 2458 --- [scanner-112152523-1,80bfb11a779d0060,b8c752fb1c08eb19] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:13:59.249 INFO 2458 --- [alarm-detect-13,80bfb11a779d0060,3f08ff654d905c85] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534810 ~ 1761534829 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534825, value=71.899)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534825, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534825=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:13:59.265 INFO 2458 --- [ocp-schedules-12,44c493b678952e94,da35ffb2217d279c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:13:59.707 INFO 2458 --- [prometheus-minute-schedule-1,c5f37f7ee6a02b85,b1ced680fe02107f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534840, interval=60 2025-10-27 11:13:59.716 INFO 2458 --- [prometheus-minute-schedule-1,c5f37f7ee6a02b85,b1ced680fe02107f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:00.126 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:00.127 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:00.531 WARN 2458 --- [ocp-async-3,f94140a97cb4043e,a6200f8212293a96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:00.531 INFO 2458 --- [ocp-async-4,b96ae761ea41bc47,2226ae1550304c57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:01.399 INFO 2458 --- [ocp-schedules-28,8da701e6c602917a,d88dbd4060959b39] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:14:01.409 INFO 2458 --- [ocp-schedules-28,8da701e6c602917a,d88dbd4060959b39] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:01.502 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:14:01.502 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.506 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:14:01.515 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.516 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.545 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.546 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.558 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.558 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.558 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:14:01.558 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.558 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.559 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.562 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:14:01.572 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.573 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.575 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.575 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.579 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.579 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.579 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.580 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.603 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.604 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.607 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.607 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.607 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.608 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.633 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.634 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.634 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.634 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.634 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:14:01.635 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.636 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.636 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.637 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.637 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.669 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.669 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.669 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.670 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.670 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.670 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.671 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.672 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.672 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.673 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.674 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.675 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.675 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.676 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.676 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.677 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.678 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:14:01.681 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:14:01.696 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.696 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.696 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.697 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:14:01.702 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.702 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.703 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.704 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:14:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:01.713725+08:00, lastProcessTime=2025-10-27T11:13:51.725350+08:00 2025-10-27 11:14:01.755 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:01.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.755 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.756 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.756 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.756 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:14:01.756 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:14:01.766 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:01.766 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:14:01.766 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:01.766 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:14:01.771 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:14:01.771 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:14:01.791 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.791 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.792 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.792 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:14:01.794 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.794 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.794 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:14:01.822 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.822 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.822 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.823 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:14:01.840 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,65eddb0474dcfaed,3c6c5e33768551fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=65eddb0474dcfaed, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:14:01.841 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,87e4051057d31bdb,58249b48b06b26d1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.40.198.13, traceId=87e4051057d31bdb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-10-27 11:14:01.844 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,45e83d4fd9e72654,711e21ed87e358dc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=45e83d4fd9e72654, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:14:01.845 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,45e83d4fd9e72654,711e21ed87e358dc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=45e83d4fd9e72654, duration=18 ms] 2025-10-27 11:14:01.848 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,5e698a503da66d29,272ccd29eeab5449] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=5e698a503da66d29, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:14:01.850 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,455833220512f57c,45368a9ea55487e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=455833220512f57c, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:14:01.850 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,455833220512f57c,45368a9ea55487e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=455833220512f57c, duration=23 ms] 2025-10-27 11:14:01.856 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.856 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.856 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.858 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:14:01.860 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,5e698a503da66d29,272ccd29eeab5449] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=5e698a503da66d29, duration=33 ms] 2025-10-27 11:14:01.881 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,87e4051057d31bdb,58249b48b06b26d1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.40.198.13, traceId=87e4051057d31bdb, duration=54 ms] 2025-10-27 11:14:01.899 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f1c2d42a2cf7596f,584d4a7ea8c2db79] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?clusterId=2&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=f1c2d42a2cf7596f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,2,] 2025-10-27 11:14:01.899 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=701395fe281f170b, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:14:01.911 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f1c2d42a2cf7596f,584d4a7ea8c2db79] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=f1c2d42a2cf7596f, duration=20 ms] 2025-10-27 11:14:01.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.921 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.923 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.924 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.924 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.924 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.925 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.926 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.926 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.926 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.927 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.928 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.928 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.929 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.930 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.930 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.930 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.930 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.931 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.932 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.932 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.932 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.933 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.933 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.933 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.933 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.935 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.936 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.936 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.936 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.937 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.938 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.938 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.939 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.939 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:14:01.940 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.940 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.940 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.941 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:14:01.941 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:01.941 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:01.942 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:01.942 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:14:01.943 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:14:01.947 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:14:01.971 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,983cf86b9ed84574,4989aae7e3fe395f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=983cf86b9ed84574, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:14:02.004 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,983cf86b9ed84574,4989aae7e3fe395f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=983cf86b9ed84574, duration=61 ms] 2025-10-27 11:14:02.029 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.029 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:14:02.037 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:01.713725+08:00 2025-10-27 11:14:02.051 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.053 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:14:02.054 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.055 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:14:02.055 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:14:02.055 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:14:02.055 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:14:02.057 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.058 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:02.098 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.099 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,701395fe281f170b,981b09039bf3b340] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=701395fe281f170b, duration=208 ms] 2025-10-27 11:14:02.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:14:02.133 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:02.133 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:14:02.133 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:02.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:14:02.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:02.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:14:02.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:14:02.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:14:02.138 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:14:02.138 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:14:02.138 INFO 2458 --- [ocp-schedules-28,8da701e6c602917a,d88dbd4060959b39] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:14:02.156 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,65eddb0474dcfaed,3c6c5e33768551fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=65eddb0474dcfaed, duration=329 ms] 2025-10-27 11:14:02.229 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,507ac4e6fd6477fb,407fbb0a8b8798f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.40.198.13, traceId=507ac4e6fd6477fb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-10-27 11:14:02.252 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,507ac4e6fd6477fb,407fbb0a8b8798f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.40.198.13, traceId=507ac4e6fd6477fb, duration=31 ms] 2025-10-27 11:14:02.509 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,c26e5a21ad36d516,5690dd3abfacfdcb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/alarmSummaries?id=2, client=10.40.198.13, traceId=c26e5a21ad36d516, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getClusterAlarmSummaries(Long), args=2,] 2025-10-27 11:14:02.515 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/summary?id=2, client=10.40.198.13, traceId=e0d1a4903c323a97, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterController.getObClusterSummary(Long), args=2,] 2025-10-27 11:14:02.515 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/stats?id=2, client=10.40.198.13, traceId=014de87cc6307951, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.getObClusterStats(Long), args=2,] 2025-10-27 11:14:02.515 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,071bf3e19480f6e2,156386cb2ca11e81] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/topSql/switch?id=2, client=10.40.198.13, traceId=071bf3e19480f6e2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.perf.ObStatCollectionConfigController.queryTopSqlSwitch(Long), args=2,] 2025-10-27 11:14:02.516 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/serverStats?id=2, client=10.40.198.13, traceId=be7ee23dbd9e6a4e, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterStatsController.listObServerStats(Long), args=2,] 2025-10-27 11:14:02.517 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,fda29e4d59088328,8e8bc5343b99195b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/eventMeta?id=2, client=10.40.198.13, traceId=fda29e4d59088328, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEventMeta(Long), args=2,] 2025-10-27 11:14:02.531 INFO 2458 --- [ocp-async-9,51364139356d8085,f247f7cd070d9733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:02.531 WARN 2458 --- [ocp-async-7,d2fb5292dfe25da8,8f6d67252775d122] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:02.534 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,fda29e4d59088328,8e8bc5343b99195b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/eventMeta client=10.40.198.13, traceId=fda29e4d59088328, duration=39 ms] 2025-10-27 11:14:02.586 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,c26e5a21ad36d516,5690dd3abfacfdcb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/alarmSummaries client=10.40.198.13, traceId=c26e5a21ad36d516, duration=90 ms] 2025-10-27 11:14:02.597 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,071bf3e19480f6e2,156386cb2ca11e81] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/topSql/switch client=10.40.198.13, traceId=071bf3e19480f6e2, duration=102 ms] 2025-10-27 11:14:02.601 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:14:02.606 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.606 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.606 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:14:02.609 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:14:02.609 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.610 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:14:02.610 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.611 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:14:02.613 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.613 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:14:02.617 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,87c805e0eea9e900,f209f5d8548dac83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/tenants/binlogService?id=2, client=10.40.198.13, traceId=87c805e0eea9e900, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObTenantBinlogController.listObClusterRelatedBinlogService(Long), args=2,] 2025-10-27 11:14:02.617 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,014de87cc6307951,8b6b3216d9cee0be] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/stats client=10.40.198.13, traceId=014de87cc6307951, duration=122 ms] 2025-10-27 11:14:02.620 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,be7ee23dbd9e6a4e,bef5afbe13cf04a8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/serverStats client=10.40.198.13, traceId=be7ee23dbd9e6a4e, duration=125 ms] 2025-10-27 11:14:02.671 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,87c805e0eea9e900,f209f5d8548dac83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/tenants/binlogService client=10.40.198.13, traceId=87c805e0eea9e900, duration=64 ms] 2025-10-27 11:14:02.683 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0d1320c2d8cc3298,aa624179f0a3709d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/events?startTime=2025-10-20T11%3A14%3A02%2B08%3A00&endTime=2025-10-27T11%3A14%3A02%2B08%3A00&page=1&size=7, client=10.40.198.13, traceId=0d1320c2d8cc3298, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterEventController.listObClusterEvents(Long,OffsetDateTime,OffsetDateTime,List,List,Pageable), args=2,2025-10-20T11:14:02+08:00,2025-10-27T11:14:02+08:00,Page request [number: 0, size 7, sort: timestamp: DESC],] 2025-10-27 11:14:02.694 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.694 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:14:02.707 INFO 2458 --- [ocp-schedules-25,0b6c779167140094,61ec47c953ca5df9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:14:02.707 INFO 2458 --- [prometheus-minute-schedule-1,b1d66505bdebfc13,ba197451ed72b77e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:02.714 INFO 2458 --- [ocp-schedules-25,0b6c779167140094,61ec47c953ca5df9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:02.716 INFO 2458 --- [prometheus-minute-schedule-1,b1d66505bdebfc13,ba197451ed72b77e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:02.743 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.746 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:14:02.747 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.748 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from __all_virtual_sys_parameter_stat where name='system_memory' 2025-10-27 11:14:02.750 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.751 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ svr_ip, value from GV$SYSSTAT where stat_id=140015 and con_id=1 2025-10-27 11:14:02.772 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0d1320c2d8cc3298,aa624179f0a3709d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/events client=10.40.198.13, traceId=0d1320c2d8cc3298, duration=103 ms] 2025-10-27 11:14:02.773 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.773 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:14:02.793 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,e0d1a4903c323a97,72f0e20c948b0805] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/summary client=10.40.198.13, traceId=e0d1a4903c323a97, duration=298 ms] 2025-10-27 11:14:02.799 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:02.799 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:02.799 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:14:02.799 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:14:02.800 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:02.800 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:14:02.821 INFO 2458 --- [ocp-schedules-25,0b6c779167140094,61ec47c953ca5df9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:14:02.888 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,cd23c1db21837502,e0943ce597490428] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=cd23c1db21837502, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:14:02.907 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,cd23c1db21837502,e0943ce597490428] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=cd23c1db21837502, duration=25 ms] 2025-10-27 11:14:03.127 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:03.128 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:03.824 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,1bdef028e87a4bf3,a080fd77e19d7ab0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=1bdef028e87a4bf3, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:14:03.824 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,1bdef028e87a4bf3,a080fd77e19d7ab0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=1bdef028e87a4bf3, duration=10 ms] 2025-10-27 11:14:03.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,c69d87e6bfce9672,899c232cad7452d8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=c69d87e6bfce9672, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:14:03.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,c69d87e6bfce9672,899c232cad7452d8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=c69d87e6bfce9672, duration=12 ms] 2025-10-27 11:14:03.829 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,33f19d7eb4ebd4f4,d67320f3e043f890] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=33f19d7eb4ebd4f4, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:14:03.829 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,33f19d7eb4ebd4f4,d67320f3e043f890] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=33f19d7eb4ebd4f4, duration=15 ms] 2025-10-27 11:14:03.852 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:14:03.852 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:14:03.852 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:14:04.051 INFO 2458 --- [ocp-schedules-1,74563dd7f207235c,5eb81a1bb4b2a258] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:04.056 INFO 2458 --- [ocp-schedules-1,74563dd7f207235c,5eb81a1bb4b2a258] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:04.525 INFO 2458 --- [ocp-schedules-27,545581156b00aad6,e6a779a28acfa505] c.o.o.a.service.AlarmTargetServiceImpl : alarm targets changed, pre targets ,,C_binlogtest2-4,C_binlogtest2-4/C_myoceanbase2_1760510188/T_test2,C_binlogtest2-4/H_10.40.65.151,C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY,C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188/H_10.40.66.107,C_obproxyTest-1,C_obproxyTest-1/H_10.40.67.22,C_obproxyTest-1/H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22, current targets ,,,C_binlogtest2-4,C_binlogtest2-4/H_10.40.65.151,C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY,C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188/H_10.40.66.107,C_obproxyTest-1,C_obproxyTest-1/H_10.40.67.22,C_obproxyTest-1/H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22,H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:04.531 WARN 2458 --- [ocp-async-1,9b072f81c72fe4c0,b3e26931de8df7a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:04.531 INFO 2458 --- [ocp-async-2,1d02d82ebf88bff1,18f29ab39dc0cec8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:05.707 INFO 2458 --- [prometheus-minute-schedule-1,b31a431a1333df3d,fe0a4b17cd1e1e2f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:05.715 INFO 2458 --- [prometheus-minute-schedule-1,b31a431a1333df3d,fe0a4b17cd1e1e2f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:14:06.055 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:14:06.097 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:14:06.122 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:14:06.123 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:14:06.123 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:14:06.123 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:14:06.125 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:14:06.126 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:06.127 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:14:06.128 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:06.129 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:06.129 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:06.129 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:14:06.150 INFO 2458 --- [ocp-schedules-10,f701384dce3686df,cf39307b63614ea4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:06.254 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:11:30, start=2025-10-27T11:07:30, end=2025-10-27T11:10:00, updates=313, granule=120000ms, delay=240000ms 2025-10-27 11:14:06.266 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:11:30, target=2025-10-27T11:10:00, granule=120000ms, delay=90000ms) 2025-10-27 11:14:06.266 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:10:00, target=2025-10-27T11:02:30, granule=600000ms, delay=450000ms) 2025-10-27 11:14:06.266 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=268ms, period=120000ms 2025-10-27 11:14:06.531 INFO 2458 --- [ocp-async-3,1cd2b8af55b35785,f0c8a67af49cffa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:06.531 WARN 2458 --- [ocp-async-4,32cd42f2fc9afd82,9fd3574d5c9a57ce] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:06.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffa10ba6fbe7de8c,c46cbeab56fa09b8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=ffa10ba6fbe7de8c, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:14:06.821 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffa10ba6fbe7de8c,c46cbeab56fa09b8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=ffa10ba6fbe7de8c, duration=11 ms] 2025-10-27 11:14:06.949 INFO 2458 --- [ocp-schedules-9,9f5f486aff5ea1b5,a475248a998c0fc0] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:14:06.956 INFO 2458 --- [ocp-schedules-9,9f5f486aff5ea1b5,a475248a998c0fc0] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:06.956 INFO 2458 --- [ocp-schedules-9,9f5f486aff5ea1b5,a475248a998c0fc0] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:08.515 INFO 2458 --- [ocp-async-7,43923e78cc7c5c4b,dfd2c6ae74951453] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:08.531 INFO 2458 --- [ocp-async-5,9a807d09fbaa17c3,202619787fdcdf36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:08.531 WARN 2458 --- [ocp-async-8,c2fd2e3603c89d0e,91cd51b4e033c394] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:08.707 INFO 2458 --- [prometheus-minute-schedule-1,79e526fec0445e18,3e0732c833534336] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:08.715 INFO 2458 --- [prometheus-minute-schedule-1,79e526fec0445e18,3e0732c833534336] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:09.129 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:09.130 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:09.255 INFO 2458 --- [ocp-schedules-28,e59fa0ab79b48c1e,9925c09441f2c0dd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:14:09.281 WARN 2458 --- [scanner-112152682-1,e59fa0ab79b48c1e,bb3d37caa424da01] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:14:09.305 INFO 2458 --- [alarm-detect-10,e59fa0ab79b48c1e,23fa32a8ea36d1fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:09.310 INFO 2458 --- [alarm-detect-10,e59fa0ab79b48c1e,23fa32a8ea36d1fc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:09.319 INFO 2458 --- [alarm-detect-10,e59fa0ab79b48c1e,6fd366236a6f9f06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:09.321 INFO 2458 --- [alarm-detect-8,e59fa0ab79b48c1e,edfb7c8b2f88fbe5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:09.324 INFO 2458 --- [alarm-detect-10,e59fa0ab79b48c1e,6fd366236a6f9f06] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:09.328 INFO 2458 --- [alarm-detect-8,e59fa0ab79b48c1e,edfb7c8b2f88fbe5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:09.474 INFO 2458 --- [alarm-detect-8,e59fa0ab79b48c1e,2976193619dcbfba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:09.478 INFO 2458 --- [alarm-detect-8,e59fa0ab79b48c1e,2976193619dcbfba] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:10.013 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:14:10.013 INFO 2458 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:14:10.045 INFO 2458 --- [Check obproxy1,f9d2ae2ff4a79581,51e345b59dde3003] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:14:10.045 INFO 2458 --- [Check obproxy1,f9d2ae2ff4a79581,51e345b59dde3003] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:14:10.049 INFO 2458 --- [Collect obproxy1,f37a444c68bbd6e2,977ad5afb71d7604] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:14:10.049 INFO 2458 --- [Collect obproxy1,f37a444c68bbd6e2,977ad5afb71d7604] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:14:10.073 INFO 2458 --- [Check obproxy1,f9d2ae2ff4a79581,51e345b59dde3003] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:14:10.076 INFO 2458 --- [Check obproxy1,f9d2ae2ff4a79581,51e345b59dde3003] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:14:10.076 INFO 2458 --- [Check obproxy1,f9d2ae2ff4a79581,51e345b59dde3003] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:14:10.076 INFO 2458 --- [Collect obproxy1,f37a444c68bbd6e2,977ad5afb71d7604] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:14:10.079 INFO 2458 --- [Collect obproxy1,f37a444c68bbd6e2,977ad5afb71d7604] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:14:10.082 INFO 2458 --- [Collect obproxy1,f37a444c68bbd6e2,977ad5afb71d7604] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:14:10.176 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:14:10.531 WARN 2458 --- [ocp-async-2,98f04ac3cb1f7e99,e2e160079dd26956] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:10.532 INFO 2458 --- [ocp-async-6,8a74e4dabe3d1c63,79a9214f41ac6204] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:10.888 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534780, nodeCount=47, affectRows=47 2025-10-27 11:14:10.909 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534780, nodeCount=47, affectRows=47 2025-10-27 11:14:11.707 INFO 2458 --- [prometheus-minute-schedule-1,ee34df9e9642a6ca,e3f16dd9e70c3277] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:11.714 INFO 2458 --- [prometheus-minute-schedule-1,ee34df9e9642a6ca,e3f16dd9e70c3277] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:11.715975+08:00, lastProcessTime=2025-10-27T11:14:01.713725+08:00 2025-10-27 11:14:11.944 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:11.715975+08:00 2025-10-27 11:14:12.130 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:12.131 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:12.532 WARN 2458 --- [ocp-async-4,ee93edee84a1f86a,1b5c0ebf38df5c0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:12.532 INFO 2458 --- [ocp-async-4,fc36c17084f66e2f,bcd5be71c8c31003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:13.766 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534780, nodeCount=723, affectRows=723 2025-10-27 11:14:14.064 INFO 2458 --- [ocp-schedules-6,cc014c9174d9dd93,50e46fc1e8b35fe6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:14.068 INFO 2458 --- [ocp-schedules-6,cc014c9174d9dd93,50e46fc1e8b35fe6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:14.532 INFO 2458 --- [ocp-async-7,8fb6bf905ce1c444,cc85c05ce43a92cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:14.532 WARN 2458 --- [ocp-async-8,61ede61d3300c3ed,18c384165b85ed47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:14.707 INFO 2458 --- [prometheus-minute-schedule-1,9e7d9f56bd67cdce,3dd4cf9d53eebbf8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:14.714 INFO 2458 --- [prometheus-minute-schedule-1,9e7d9f56bd67cdce,3dd4cf9d53eebbf8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:14.790 INFO 2458 --- [ocp-schedules-11,31c81e515a5e8cfa,668beceeb71c2f87] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:14:14.790+08:00, clusters=[1, 2] 2025-10-27 11:14:14.793 INFO 2458 --- [ocp-schedules-11,31c81e515a5e8cfa,668beceeb71c2f87] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:14:14.795 INFO 2458 --- [ocp-schedules-11,31c81e515a5e8cfa,668beceeb71c2f87] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:14:14.864 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=1000, affectRows=1000 2025-10-27 11:14:14.868 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=994, validCount=992 2025-10-27 11:14:14.928 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=1000, affectRows=1000 2025-10-27 11:14:14.933 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=999, validCount=995 2025-10-27 11:14:14.947 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570098) Duplicate entry '835014129065861120-1761534720' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570098) Duplicate entry '835014129065861120-1761534720' for key 'PRIMARY' 2025-10-27 11:14:15.131 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:15.132 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:15.252 INFO 2458 --- [ocp-schedules-28,abd58fc6734e10db,7371480a0deda41d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:14:15.252 INFO 2458 --- [ocp-schedules-28,abd58fc6734e10db,7371480a0deda41d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:14:15.304 INFO 2458 --- [Collect all tenants major compactions1,abd58fc6734e10db,9c2cf62b6df5ad62] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:14:15.304 INFO 2458 --- [Collect all tenants major compactions2,abd58fc6734e10db,6f22fa672eb0210d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:14:15.305 INFO 2458 --- [Collect all tenants major compactions5,abd58fc6734e10db,d87a5f439f190094] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:14:15.305 INFO 2458 --- [Collect all tenants major compactions3,abd58fc6734e10db,b5ea28e5e6921bfd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:14:15.305 INFO 2458 --- [Collect all tenants major compactions4,abd58fc6734e10db,f8b29d35985e389a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:14:15.306 INFO 2458 --- [Collect all tenants major compactions6,abd58fc6734e10db,b9adc3c227f9dec6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:14:15.355 INFO 2458 --- [Collect all tenants major compactions6,abd58fc6734e10db,b9adc3c227f9dec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.357 INFO 2458 --- [Collect all tenants major compactions6,abd58fc6734e10db,b9adc3c227f9dec6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:14:15.361 INFO 2458 --- [Collect all tenants major compactions6,abd58fc6734e10db,b9adc3c227f9dec6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:14:15.363 INFO 2458 --- [Collect all tenants major compactions5,abd58fc6734e10db,d87a5f439f190094] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.364 INFO 2458 --- [Collect all tenants major compactions5,abd58fc6734e10db,d87a5f439f190094] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:14:15.367 INFO 2458 --- [Collect all tenants major compactions5,abd58fc6734e10db,d87a5f439f190094] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:14:15.368 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=1000, affectRows=1000 2025-10-27 11:14:15.373 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=996, validCount=992 2025-10-27 11:14:15.391 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=84, affectRows=84 2025-10-27 11:14:15.391 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=82, validCount=82 2025-10-27 11:14:15.394 INFO 2458 --- [Collect all tenants major compactions2,abd58fc6734e10db,6f22fa672eb0210d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.394 INFO 2458 --- [Collect all tenants major compactions1,abd58fc6734e10db,9c2cf62b6df5ad62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.394 INFO 2458 --- [Collect all tenants major compactions4,abd58fc6734e10db,f8b29d35985e389a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.394 INFO 2458 --- [Collect all tenants major compactions3,abd58fc6734e10db,b5ea28e5e6921bfd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:15.395 INFO 2458 --- [Collect all tenants major compactions2,abd58fc6734e10db,6f22fa672eb0210d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:14:15.395 INFO 2458 --- [Collect all tenants major compactions4,abd58fc6734e10db,f8b29d35985e389a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:14:15.396 INFO 2458 --- [Collect all tenants major compactions1,abd58fc6734e10db,9c2cf62b6df5ad62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:14:15.396 INFO 2458 --- [Collect all tenants major compactions3,abd58fc6734e10db,b5ea28e5e6921bfd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:14:15.398 INFO 2458 --- [Collect all tenants major compactions2,abd58fc6734e10db,6f22fa672eb0210d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:14:15.399 INFO 2458 --- [Collect all tenants major compactions4,abd58fc6734e10db,f8b29d35985e389a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:14:15.399 INFO 2458 --- [Collect all tenants major compactions1,abd58fc6734e10db,9c2cf62b6df5ad62] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:14:15.400 INFO 2458 --- [Collect all tenants major compactions3,abd58fc6734e10db,b5ea28e5e6921bfd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:14:15.401 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=1000, affectRows=1000 2025-10-27 11:14:15.406 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=1000, validCount=996 2025-10-27 11:14:15.406 INFO 2458 --- [ocp-schedules-28,abd58fc6734e10db,7371480a0deda41d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:14:15.740 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73} 2025-10-27 11:14:15.749 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null 2025-10-27 11:14:15.750 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:15.752 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:15.752 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:14:15.752 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:15.755 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:15.764 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:15.794 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:14:15.794 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:15.909 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448455909000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:14:15.909Z, endTime=2025-10-27T03:12:15.730Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:14:15.912 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448455912000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=DC0ECB93D5AA3E26623B4E6CF05B6E74, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:14:15.912Z, endTime=2025-10-27T03:12:15.730Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:14:15.922 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448455922000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=0D783B41584D45C92EB73F0272E61D00, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:14:15.922Z, endTime=2025-10-27T03:12:15.730Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:14:15.936 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:14:15.936 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:14:15.959 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 150 2025-10-27 11:14:15.959 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:14:15.959 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 149 2025-10-27 11:14:15.959 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:14:15.973 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:15.973 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:15.982 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:14:15.982 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:14:15.988 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:15.988 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=DC0ECB93D5AA3E26623B4E6CF05B6E74, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:15.992 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:15.992 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=0D783B41584D45C92EB73F0272E61D00, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:16.001 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records, use plan hash 2025-10-27 11:14:16.001 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query: data skipped cause only one sample (VUlELTEtMTQ5MDQ2LTE3NjE0ODY5OTQzMjg5MDY=, VUlELTEtMTQ5MDQ2LTE3NjE0ODY5OTQzMjg5MDY=) 2025-10-27 11:14:16.007 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records, use plan hash 2025-10-27 11:14:16.007 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query: data skipped cause only one sample (VUlELTEtMTQ5MDQ3LTE3NjE0ODY5OTQzMzc3MDg=, VUlELTEtMTQ5MDQ3LTE3NjE0ODY5OTQzMzc3MDg=) 2025-10-27 11:14:16.011 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:16.011 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:16.016 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:14:16.016 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:14:15.909, end=2025-10-27T11:12:15.73}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:14:16.026 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:14:16.026 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:14:16.051 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:16.051 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=DC0ECB93D5AA3E26623B4E6CF05B6E74, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:16.051 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:14:16.051 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534735730000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=0D783B41584D45C92EB73F0272E61D00, sqlIdList=null, dynamicSql=false) 2025-10-27 11:14:16.060 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:14:16.062 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:14:16.093 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:14:15.922, end=2025-10-27T11:12:15.73}, clusterId=2, tenantId=5, sqlId=0D783B41584D45C92EB73F0272E61D00, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:14:16.106 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:14:15.912, end=2025-10-27T11:12:15.73}, clusterId=2, tenantId=5, sqlId=DC0ECB93D5AA3E26623B4E6CF05B6E74, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:14:16.106 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:16.108 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:16.108 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:14:16.108 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:16.111 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:16.112 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:16.112 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:14:16.112 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:16.115 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:16.118 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:16.118 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:14:16.118 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:16.123 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:14:16.124 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:15.276, end=2025-10-27T11:12:15.73},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:14:16.124 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:14:16.125 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:14:16.127 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:14:16.156 INFO 2458 --- [ocp-schedules-30,ee4e64a61a869f36,41bb1eabb6f3fb45] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:16.532 WARN 2458 --- [ocp-async-5,710d6fa30475a71d,a42bccc8236f48e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:16.532 INFO 2458 --- [ocp-async-2,dc75b067c8955b2f,9b9ae26048c28f37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:17.290 INFO 2458 --- [ocp-schedules-3,80b06445f8730a8c,e76e23a0ad72be03] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:14:17.291 INFO 2458 --- [ocp-schedules-3,80b06445f8730a8c,e76e23a0ad72be03] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:14:17.298 INFO 2458 --- [ocp-schedules-3,80b06445f8730a8c,e76e23a0ad72be03] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:14:17.298 INFO 2458 --- [ocp-schedules-3,80b06445f8730a8c,e76e23a0ad72be03] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:14:17.707 INFO 2458 --- [prometheus-minute-schedule-1,e0cabf7beb271e13,cfe84bc49049740c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:17.719 INFO 2458 --- [prometheus-minute-schedule-1,e0cabf7beb271e13,cfe84bc49049740c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:18.132 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:18.133 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:18.515 INFO 2458 --- [ocp-async-2,c1d1764df067c145,2d1dfd10c85f4f2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:18.532 WARN 2458 --- [ocp-async-3,b356a11aa8b51191,adf0f41adf974ca0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:18.532 INFO 2458 --- [ocp-async-6,b7dc6335dd80e3f6,4b5ec48c917e150d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:18.855 INFO 2458 --- [ocp-schedules-1,e4fe7ca08d54b92e,3b732fefe663ed3e] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:18.855 INFO 2458 --- [ocp-schedules-1,e4fe7ca08d54b92e,3b732fefe663ed3e] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:14:19.194 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534780, nodeCount=1000, affectRows=1000 2025-10-27 11:14:19.198 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534780, nodeCount=996, validCount=983 2025-10-27 11:14:19.512 INFO 2458 --- [ocp-schedules-20,bae09df0e94823b8,3c1725eb7781b8ea] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:14:19.538 WARN 2458 --- [scanner-112154496-1,bae09df0e94823b8,22367b6ce9a006eb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:14:19.588 INFO 2458 --- [ocp-schedules-25,1662278d6a8ec8c9,3a78516a0f70ae5b] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:14:19.650 INFO 2458 --- [alarm-detect-20,bae09df0e94823b8,bd34251823d63103] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534825 ~ 1761534849 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534840, value=66.69)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534840, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534840=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:14:20.532 WARN 2458 --- [ocp-async-2,7fbcc7f77db8b93a,a7fd8904310f97d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:20.532 INFO 2458 --- [ocp-async-9,bac1dd98589899f9,5076b92af28640a4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:20.707 INFO 2458 --- [prometheus-minute-schedule-1,2fef02873246de29,e82792ead054ee49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:20.717 INFO 2458 --- [prometheus-minute-schedule-1,2fef02873246de29,e82792ead054ee49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:21.133 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:21.133 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:21.410 INFO 2458 --- [ocp-schedules-14,84047508d50c0159,518581e2c9aefe3d] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:21.410 INFO 2458 --- [ocp-schedules-12,eb07b5a50772f23a,4d9515e66b9da763] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:21.410 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:14:21.410 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:14:21.411 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:14:21.427 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:21.428 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:21.428 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:21.428 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:21.428 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:21.429 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:21.429 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:21.429 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:21.432 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:21.432 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:21.432 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:21.432 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:21.433 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:21.434 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:21.434 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:21.434 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:21.434 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:21.434 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:21.436 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:21.437 INFO 2458 --- [ocp-schedules-7,b2df033ee84abff6,d3dd45eba32eb45f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:21.438 INFO 2458 --- [host-check-clockdiff1,b2df033ee84abff6,b59dddfa5127a58b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:14:21.438 INFO 2458 --- [host-check-clockdiff2,b2df033ee84abff6,fd8f29dee643cbff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:14:21.438 INFO 2458 --- [host-check-clockdiff3,b2df033ee84abff6,ab8312413bae3f60] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:14:21.438 INFO 2458 --- [host-check-clockdiff4,b2df033ee84abff6,d1b0af8593ecfde6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:14:21.438 INFO 2458 --- [host-check-clockdiff5,b2df033ee84abff6,44b1263115fa953d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:14:21.439 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:21.442 INFO 2458 --- [ocp-schedules-27,817aab1790dc1306,448a8f5eb1e98aef] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:14:21.463 INFO 2458 --- [host-check-clockdiff5,b2df033ee84abff6,44b1263115fa953d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534861 0 0 , error= 2025-10-27 11:14:21.463 INFO 2458 --- [host-check-clockdiff2,b2df033ee84abff6,fd8f29dee643cbff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534861 -1 -1 , error= 2025-10-27 11:14:21.464 INFO 2458 --- [host-check-clockdiff1,b2df033ee84abff6,b59dddfa5127a58b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534861 0 0 , error= 2025-10-27 11:14:21.464 INFO 2458 --- [host-check-clockdiff3,b2df033ee84abff6,ab8312413bae3f60] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534861 -1 -1 , error= 2025-10-27 11:14:21.507 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:14:21.507 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:14:21.507 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:14:21.509 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:14:21.509024950+08:00 2025-10-27 11:14:21.509 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:14:21.509540921+08:00 2025-10-27 11:14:21.509 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:14:21.509 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:14:21.509+08:00 2025-10-27 11:14:21.510 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:14:21.509+08:00 2025-10-27 11:14:21.512 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:14:21.512 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:14:21.512 INFO 2458 --- [host-check-agent-status3,eb07b5a50772f23a,3d34697eb1f945a8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:14:21.513 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:14:21.513 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:14:21.513 INFO 2458 --- [host-check-agent-status4,eb07b5a50772f23a,e2072e9b0631f4dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:14:21.515 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:14:21.517 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:14:21.509+08:00 and status AVAILABLE 2025-10-27 11:14:21.517 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:14:21.517 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:14:21.517150627+08:00 2025-10-27 11:14:21.518 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:14:21.516434984+08:00 2025-10-27 11:14:21.518 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:14:21.518+08:00 2025-10-27 11:14:21.519 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:14:21.518+08:00 2025-10-27 11:14:21.520 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:14:21.520 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:14:21.520 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:14:21.509+08:00 and status AVAILABLE 2025-10-27 11:14:21.522 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:14:21.522 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:14:21.522 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:14:21.522 INFO 2458 --- [host-check-agent-status5,eb07b5a50772f23a,c335c869f2d8bb7d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:14:21.523 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:14:21.523 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:14:21.523 INFO 2458 --- [host-check-agent-status2,eb07b5a50772f23a,a6a7dedef9376e1b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:14:21.524 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:14:21.524 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:14:21.524 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:14:21.524047069+08:00 2025-10-27 11:14:21.525 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:14:21.525 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:14:21.524+08:00 2025-10-27 11:14:21.526 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:14:21.518+08:00 and status AVAILABLE 2025-10-27 11:14:21.526 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:14:21.518+08:00 and status AVAILABLE 2025-10-27 11:14:21.529 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:14:21.529 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:14:21.529 INFO 2458 --- [host-check-agent-status1,eb07b5a50772f23a,25451467aae1fa43] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:14:21.531 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:14:21.532 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:14:21.524+08:00 and status AVAILABLE 2025-10-27 11:14:21.532 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:14:21.536 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:14:21.542 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:21.545 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:21.545 INFO 2458 --- [ocp-schedules-30,c6273e5d86ca1d3a,8f2d3ee1034365e6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:14:21.546 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:21.549 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:21.549 INFO 2458 --- [ocp-schedules-28,2b72c2c3db1228df,c09a55e4ef24d449] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:14:21.550 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:21.550 INFO 2458 --- [ocp-schedules-30,c6273e5d86ca1d3a,8f2d3ee1034365e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:21.553 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:21.553 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:21.554 INFO 2458 --- [ocp-schedules-22,aba415b6b234620c,6fd9ca8ca42a464d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:14:21.554 INFO 2458 --- [ocp-schedules-28,2b72c2c3db1228df,c09a55e4ef24d449] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:21.555 INFO 2458 --- [ocp-schedules-28,2b72c2c3db1228df,c09a55e4ef24d449] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:14:21.556 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:21.556 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:21.558 INFO 2458 --- [ocp-schedules-22,aba415b6b234620c,6fd9ca8ca42a464d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:21.559 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:21.566 INFO 2458 --- [ocp-schedules-10,eb28c5f7d9b87a51,0b91c7fe9be04c79] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:14:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:21.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:14:21.609 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:21.609 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:21.610 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.611 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:14:21.617 INFO 2458 --- [ocp-schedules-30,c6273e5d86ca1d3a,8f2d3ee1034365e6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:14:21.617 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:21.620 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:21.625 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:21.626 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:14:21.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:21.636 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:14:21.638 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:14:21.639 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:14:21.647 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:14:21.656 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:14:21.665 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.666 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:21.667 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:14:21.668 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:14:21.674 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:14:21.678 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:14:21.689 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:21.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:14:21.704 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:21.710 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:21.713 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:21.713 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:21.712613+08:00, lastProcessTime=2025-10-27T11:14:11.715975+08:00 2025-10-27 11:14:21.716 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:21.722 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:14:21.728 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:14:21.731 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:14:21.741 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:14:21.747 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:14:21.757 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:21.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:21.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.760 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:14:21.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:14:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:21.764 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:21.765 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.765 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:14:21.772 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:14:21.783 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:14:21.787 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:14:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:21.804 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:14:21.818 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:21.828 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:21.830 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:21.832 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:21.835 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:14:21.838 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:21.842 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:14:21.847 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:21.850 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:14:21.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:21.861 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.861 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:21.862 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:14:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:21.865 INFO 2458 --- [host-check-host-status5,84047508d50c0159,fa32a794984828ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:14:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.866 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:21.867 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:14:21.868 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:14:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.872 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:14:21.873 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:21.874 INFO 2458 --- [host-check-host-status4,84047508d50c0159,e4c7d8bb3a9c3af6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:14:21.875 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:14:21.883 INFO 2458 --- [host-check-host-status2,84047508d50c0159,1ef62e5658f27af4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:14:21.886 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:21.890 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:21.902 INFO 2458 --- [host-check-host-status1,84047508d50c0159,f8d50886f031c14f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:14:21.902 INFO 2458 --- [host-check-host-status3,84047508d50c0159,185e26d1700ab0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:14:21.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:21.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:21.951 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:14:21.957 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:21.957 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:21.993 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:14:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:14:21.996 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:21.997 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:21.997 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:21.997 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:14:22.015 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:14:22.040 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:22.041 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:22.041 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:22.041 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:22.042 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:14:22.064 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:14:22.064 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:14:22.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:22.074 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:22.113 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:22.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:14:22.166 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:14:22.187 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:22.191 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:22.242 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:14:22.319 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:14:22.319 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:22.320 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:14:22.322 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:22.323 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:14:22.357 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:14:22.361 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:21.712613+08:00 2025-10-27 11:14:22.394 INFO 2458 --- [host-check-clockdiff4,b2df033ee84abff6,d1b0af8593ecfde6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534862 0 0 , error= 2025-10-27 11:14:22.401 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:22.402 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:22.402 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:14:22.461 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:14:22.461 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:14:22.461 INFO 2458 --- [ocp-schedules-22,aba415b6b234620c,6fd9ca8ca42a464d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:14:22.532 INFO 2458 --- [ocp-async-9,bb1e8cab9656d4e9,af8c3ae7281f2a62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:22.532 WARN 2458 --- [ocp-async-1,169366a323cd7531,fb58282a9b5bc8d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:23.707 INFO 2458 --- [prometheus-minute-schedule-1,1e981a90a3a8cd4f,be4e4d4b2769e704] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:23.718 INFO 2458 --- [prometheus-minute-schedule-1,1e981a90a3a8cd4f,be4e4d4b2769e704] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:23.903 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,6e275763b6da8424,3b246f2b2e62a8e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=6e275763b6da8424, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:14:23.903 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,6e275763b6da8424,3b246f2b2e62a8e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=6e275763b6da8424, duration=1 ms] 2025-10-27 11:14:24.077 INFO 2458 --- [ocp-schedules-5,8398a137dc09fcba,5d57b70ae52a3485] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:24.081 INFO 2458 --- [ocp-schedules-5,8398a137dc09fcba,5d57b70ae52a3485] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:24.134 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:24.135 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:24.533 WARN 2458 --- [ocp-async-3,9196ed369adfa7d1,037ca017f7e4a1de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:24.533 INFO 2458 --- [ocp-async-4,15742902e6e4fe3f,51dd6660f9cd7d55] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:26.165 INFO 2458 --- [ocp-schedules-26,ca6db95713d03842,fb4c088abafcefb6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:26.314 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,18f7aee5daff8eb6,a6a2be46bcddc564] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.40.198.13, traceId=18f7aee5daff8eb6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:14:26.315 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,7762a2202608c711,1ceabe9e7adb4406] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?withInstallationData=true&type=OB_SERVER_INSTALL_PACKAGE, client=10.40.198.13, traceId=7762a2202608c711, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=[OB_SERVER_INSTALL_PACKAGE],true,Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:14:26.328 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,18f7aee5daff8eb6,a6a2be46bcddc564] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.40.198.13, traceId=18f7aee5daff8eb6, duration=42 ms] 2025-10-27 11:14:26.332 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1b13ef3c3b06b49e,7c95c44a76d7403c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.40.198.13, traceId=1b13ef3c3b06b49e, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-10-27 11:14:26.335 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters/2/startupParameters?id=2, client=10.40.198.13, traceId=7644a7d57eb72158, method=IterableResponse com.oceanbase.ocp.server.common.controller.ObClusterParameterController.listObClusterStartupParameters(Long), args=2,] 2025-10-27 11:14:26.336 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5692554e95a93888,9602cbf7defa553d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE&status=ONLINE&excludedServiceType=OB_CLUSTER&excludedServiceType=BACKUP_AGENT&excludedServiceType=RESTORE_AGENT&excludedServiceType=BACKUP_RESTORE_AGENT&excludedServiceType=ARBITRATION&excludedServiceType=BINLOG_SERVICE, client=10.40.198.13, traceId=5692554e95a93888, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE, ONLINE],[OB_CLUSTER, BACKUP_AGENT, RESTORE_AGENT, BACKUP_RESTORE_AGENT, ARBITRATION, BINLOG_SERVICE],Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:14:26.338 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,2177a9d1b343b1c1,6639393f18ee83b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?null, client=10.40.198.13, traceId=2177a9d1b343b1c1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:14:26.340 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,2177a9d1b343b1c1,6639393f18ee83b0] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-10-27 11:14:26.354 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1b13ef3c3b06b49e,7c95c44a76d7403c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.40.198.13, traceId=1b13ef3c3b06b49e, duration=66 ms] 2025-10-27 11:14:26.357 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-10,5692554e95a93888,9602cbf7defa553d] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-10-27 11:14:26.384 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5692554e95a93888,9602cbf7defa553d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.40.198.13, traceId=5692554e95a93888, duration=96 ms] 2025-10-27 11:14:26.396 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,8cde7de3ac940939,2b5ebf05cb0ad3ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.40.198.13, traceId=8cde7de3ac940939, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-10-27 11:14:26.418 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,8cde7de3ac940939,2b5ebf05cb0ad3ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.40.198.13, traceId=8cde7de3ac940939, duration=30 ms] 2025-10-27 11:14:26.435 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:26.436 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ version() 2025-10-27 11:14:26.439 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:26.439 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' 2025-10-27 11:14:26.505 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,2177a9d1b343b1c1,6639393f18ee83b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.40.198.13, traceId=2177a9d1b343b1c1, duration=218 ms] 2025-10-27 11:14:26.509 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:26.509 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'tenant' 2025-10-27 11:14:26.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,7762a2202608c711,1ceabe9e7adb4406] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.40.198.13, traceId=7762a2202608c711, duration=238 ms] 2025-10-27 11:14:26.533 INFO 2458 --- [ocp-async-2,03b1c56c17bd0ae6,aa12d9d255613794] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:26.533 WARN 2458 --- [ocp-async-6,438d1ff2fd28bfb3,91c28a9cc1cab779] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:26.556 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:26.558 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ t.TENANT_ID, t.TENANT_NAME, p.ZONE, p.SVR_IP, p.SVR_PORT, p.NAME, p.VALUE, p.DATA_TYPE, p.SECTION, p.EDIT_LEVEL, p.INFO FROM GV$OB_PARAMETERS p JOIN DBA_OB_TENANTS t ON p.TENANT_ID = t.TENANT_ID WHERE SCOPE = 'TENANT' AND TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:14:26.707 INFO 2458 --- [prometheus-minute-schedule-1,17b68d0085e205fc,d02447ae32823a50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:26.719 INFO 2458 --- [prometheus-minute-schedule-1,17b68d0085e205fc,d02447ae32823a50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:26.746 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,7644a7d57eb72158,6bb2cda8e00c2a35] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters/2/startupParameters client=10.40.198.13, traceId=7644a7d57eb72158, duration=458 ms] 2025-10-27 11:14:26.959 INFO 2458 --- [ocp-schedules-17,688f286e1f3b3dad,aef8c98a857e5f82] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:14:26.966 INFO 2458 --- [ocp-schedules-17,688f286e1f3b3dad,aef8c98a857e5f82] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:26.966 INFO 2458 --- [ocp-schedules-17,688f286e1f3b3dad,aef8c98a857e5f82] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:27.135 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:27.136 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:28.516 INFO 2458 --- [ocp-async-5,f3dbe39d5e96759d,1db60cd5507d2599] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:28.533 INFO 2458 --- [ocp-async-1,3c85befe3e7253e7,c66595bb2db10d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:28.533 WARN 2458 --- [ocp-async-9,caded22e4cf4d011,c30594fddfc81f45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:28.898 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:14:28.898 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:14:28.900 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:14:28.900 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839258596677459968 2025-10-27 11:14:28.903 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=839260515940638720 2025-10-27 11:14:28.903 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:14:28.903 INFO 2458 --- [ocp-schedules-19,4d2b32f3b9849fc1,6ec2cd2c2e933b36] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:14:29.361 INFO 2458 --- [ocp-schedules-4,ef247fe36512a389,f19884e51d9e3dc6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:14:29.372 INFO 2458 --- [ocp-schedules-4,ef247fe36512a389,f19884e51d9e3dc6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:14:29.372 INFO 2458 --- [ocp-schedules-4,ef247fe36512a389,f19884e51d9e3dc6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:14:29.538 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:14:29.538 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:14:29.542 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:14:29.542 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:14:29.542 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:14:29.542 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:14:29.545 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:14:29.545 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:14:29.545 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:14:29.545 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:14:29.568 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:14:29.574 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:14:29.574 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:14:29.574 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:14:29.574 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:14:29.574 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:14:29.580 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:14:29.663 INFO 2458 --- [ocp-schedules-28,3437dc93616bb89f,05323694921ce6f5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:14:29.707 INFO 2458 --- [prometheus-minute-schedule-1,50a4e716648de74b,237d7551ef784821] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:29.728 INFO 2458 --- [prometheus-minute-schedule-1,50a4e716648de74b,237d7551ef784821] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:29.791 WARN 2458 --- [scanner-112154663-1,3437dc93616bb89f,c35cfe4a46d94df1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:14:29.820 INFO 2458 --- [alarm-detect-6,3437dc93616bb89f,34f8b8250f3c3bd9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:29.825 INFO 2458 --- [alarm-detect-6,3437dc93616bb89f,34f8b8250f3c3bd9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:29.995 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:14:30.021 INFO 2458 --- [alarm-detect-18,3437dc93616bb89f,0622eaed8d173130] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:30.023 INFO 2458 --- [alarm-detect-20,3437dc93616bb89f,0e42e3a8df7d4008] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:30.027 INFO 2458 --- [alarm-detect-20,3437dc93616bb89f,0e42e3a8df7d4008] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:30.027 INFO 2458 --- [alarm-detect-18,3437dc93616bb89f,0622eaed8d173130] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:30.136 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:30.137 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:30.425 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.87938282S 2025-10-27 11:14:30.425 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:14:30.494 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:14:30.494 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:14:30.524 INFO 2458 --- [alarm-detect-14,3437dc93616bb89f,fd9b4697e9252e48] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:30.531 INFO 2458 --- [alarm-detect-14,3437dc93616bb89f,fd9b4697e9252e48] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:30.533 WARN 2458 --- [ocp-async-4,7f14cbc4ebf356ae,52c81abe9be77ef6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:30.533 INFO 2458 --- [ocp-async-8,1366b0cbe5d6380f,155a46c7e8e26778] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:30.548 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:14:30.616 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:14:30.625 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:14:30.775 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] o.s.boot.SpringApplication : Started application in 1.233 seconds (JVM running for 257339.202) 2025-10-27 11:14:30.775 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.233476471S 2025-10-27 11:14:31.180 INFO 2458 --- [ocp-schedules-29,9a0a5aa326310dd3,ec815167cc6f3f0f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:14:31.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:31.718902+08:00, lastProcessTime=2025-10-27T11:14:21.712613+08:00 2025-10-27 11:14:32.241 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:31.718902+08:00 2025-10-27 11:14:32.533 INFO 2458 --- [ocp-async-10,74a9b51a55e3c34a,603d0cb3e0f6221d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:32.533 WARN 2458 --- [ocp-async-2,32a0517410da7cf2,3feb07efed75f522] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:32.707 INFO 2458 --- [prometheus-minute-schedule-1,d91200b74b02c2bb,f2a64313dcce0203] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:32.718 INFO 2458 --- [prometheus-minute-schedule-1,d91200b74b02c2bb,f2a64313dcce0203] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:33.137 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:33.138 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:33.796 INFO 2458 --- [ocp-schedules-21,3227a8f2c11edf57,8ed6e7b9a6f2c72e] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:14:33.796 INFO 2458 --- [ocp-schedules-21,3227a8f2c11edf57,8ed6e7b9a6f2c72e] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:14:34.091 INFO 2458 --- [ocp-schedules-29,80ab1aeae2afb6a4,20233f797ac18022] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:34.095 INFO 2458 --- [ocp-schedules-29,80ab1aeae2afb6a4,20233f797ac18022] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:34.533 INFO 2458 --- [ocp-async-7,4e288c7fc30e831d,e55c37b065b62cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:34.533 WARN 2458 --- [ocp-async-5,ab5fe50a31d4ab33,c41f14799b5dc1c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:35.707 INFO 2458 --- [prometheus-minute-schedule-1,29172cfa55d69123,9844bcb8c9710163] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:35.716 INFO 2458 --- [prometheus-minute-schedule-1,29172cfa55d69123,9844bcb8c9710163] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:36.138 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:36.140 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:36.178 INFO 2458 --- [ocp-schedules-16,011c62c4009a1233,2e9396126dbb4c34] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:36.207 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_timeout, expression merge_timeout_flag == 1 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_timeout_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_timeout 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_timeout 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_drop, expression net_receive_drop_count > 100 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_receive_drop_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_drop 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_drop 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_cannot_connect_arbitration, expression arbitration_status_inactive_server_count > 0 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_status_inactive_server_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_cannot_connect_arbitration 2025-10-27 11:14:36.208 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_cannot_connect_arbitration 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_leader_absent, expression tenant_partition_leader_absent_count > 100 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_leader_absent_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_leader_absent 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_leader_absent 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cpu_percent_over_threshold 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cpu_percent_over_threshold 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_alarm_detect_timeout, expression ocp_alarm_detect_duration_seconds >= 3 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_alarm_detect_duration_seconds', labels={}}], alarmTargets 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_alarm_detect_timeout 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_alarm_detect_timeout 2025-10-27 11:14:36.209 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_retry_times_exceeded, expression storage_capacity_monitor_error_code == 1001 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_retry_times_exceeded 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_retry_times_exceeded 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ssl_certificate_is_expired, expression ssl_certificate_expired_status > 0 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ssl_certificate_expired_status', labels={}}], alarmTargets 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ssl_certificate_is_expired 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ssl_certificate_is_expired 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 95 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_log_disk_percent_over_threshold, expression ob_log_disk_usage_percent > 92 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_log_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_log_disk_percent_over_threshold 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_log_disk_percent_over_threshold 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_hwm, expression cpu_used_percent > 90 and cpu_steal < 20 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_steal', labels={}}, Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_hwm 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_hwm 2025-10-27 11:14:36.210 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_hwm, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_hwm 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_hwm 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_cannot_connected, expression obproxy_connectable == 0 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_connectable', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_cannot_connected 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_cannot_connected 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_recyclebin_disk_used_over_threshold, expression ob_database_disk_used_gb > 100 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_recyclebin_disk_used_gb', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_recyclebin_disk_used_over_threshold 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_recyclebin_disk_used_over_threshold 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for cluster_cgroup_value_inconsistent, expression cluster_cgroup_value == 1 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='cluster_cgroup_value', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=cluster_cgroup_value_inconsistent 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-cluster_cgroup_value_inconsistent 2025-10-27 11:14:36.211 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_connectable', labels={}}], alarmTargets 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_cannot_connect_to_leader 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_cannot_connect_to_leader 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_send_percent_over_threshold, expression net_send_percent > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_send_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_send_percent_over_threshold 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_send_percent_over_threshold 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_sda_ioawait, expression io_await >= 200 and io_util >= 99 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_sda_ioawait 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_sda_ioawait 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for refresh_location_cache_failed_by_metric, expression location_cache_log_count > 0 and location_cache_success_count == 0 and location_cache_sql_success_count == 0 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_location_cache_log_count', labels={}}, Metric{name='ob_cluster_location_cache_success_count', labels={}}, Metric{name='ob_cluster_location_cache_sql_success_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=refresh_location_cache_failed_by_metric 2025-10-27 11:14:36.212 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-refresh_location_cache_failed_by_metric 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_compaction_status_is_suspend, expression tenant_compaction_status == 1 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='tenant_compaction_status', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_compaction_status_is_suspend 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_compaction_status_is_suspend 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_nvme_ioawait, expression io_await > 20 and io_util >= 98 and io_qusize >= 20 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_await', labels={}}, Metric{name='ob_host_io_util', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_nvme_ioawait 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_nvme_ioawait 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ic_server_connect_failed, expression ic_server_health == 0 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ic_server_health', labels={}}], alarmTargets 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ic_server_connect_failed 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ic_server_connect_failed 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_failed, expression ob_cluster_sync == 0 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_failed 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_failed 2025-10-27 11:14:36.213 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_process_dead, expression oblb_process_exists == 0 or oblb_service_connectable == 0 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_process_exists', labels={}}, Metric{name='oblb_service_connectable', labels={}}], alarmTargets 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_process_dead 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_process_dead 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_error, expression net_traffic_err_count > 100 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_net_traffic_err_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_error 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_error 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 95 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_memstore_percent_over_threshold, expression memstore_percent > 85 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_memstore_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_memstore_percent_over_threshold 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_memstore_percent_over_threshold 2025-10-27 11:14:36.214 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_node_status_abnormal, expression binlog_node_available == 0 2025-10-27 11:14:36.215 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_node_available', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:14:36.215 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_node_status_abnormal 2025-10-27 11:14:36.215 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_node_status_abnormal 2025-10-27 11:14:36.215 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_nic_usage_over_threshold, expression oblb_net_traffic_usage > 80 and net_bandwidth_mbps > 0 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_net_traffic_usage', labels={}}, Metric{name='net_bandwidth_mbps', labels={}}], alarmTargets 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_nic_usage_over_threshold 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_nic_usage_over_threshold 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_ip_forward_invalid, expression oblb_ip_forward_status != 1 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_ip_forward_status', labels={}}], alarmTargets 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_ip_forward_invalid 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_ip_forward_invalid 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_service_unavailable, expression arbitration_service_available == 0 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_service_available', labels={}}], alarmTargets 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_service_unavailable 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_service_unavailable 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_active_memstore_percent_over_threshold, expression active_memstore_percent > 110 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_active_memstore_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_active_memstore_percent_over_threshold 2025-10-27 11:14:36.216 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_active_memstore_percent_over_threshold 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_conversion_delay, expression binlog_conversion_delay > 10000 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_conversion_delay', labels={}}], alarmTargets 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_conversion_delay 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_conversion_delay 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obagent_dead, expression host_agent_process_status == 0 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_process_status', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obagent_dead 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obagent_dead 2025-10-27 11:14:36.217 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_memory_usage_over_threshold, expression binlog_host_memory_usage > 90 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_memory_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_memory_usage_over_threshold 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_memory_usage_over_threshold 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for monagent_process_stop, expression monagent_boot_time_delta_seconds > 10 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monagent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monagent_process_stop 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monagent_process_stop 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for mgragent_process_stop, expression mgragent_boot_time_delta_seconds > 10 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='mgragent_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=mgragent_process_stop 2025-10-27 11:14:36.218 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-mgragent_process_stop 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_nic_1000m_full, expression eth_net_traffic_usage > 92 and eth_net_bandwidth_mbps < 300 and eth_net_bandwidth_mbps > 0 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_nic_1000m_full 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_nic_1000m_full 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_monitordb_disconnected, expression monitordb_connectable == 0 2025-10-27 11:14:36.219 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitordb_connectable', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_monitordb_disconnected 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_monitordb_disconnected 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_stop, expression obproxyd_boot_time_delta_seconds > 10 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_boot_time_delta_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_stop 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_stop 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_storage_usage_over_threshold, expression binlog_host_storage_usage > 90 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_storage_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_storage_usage_over_threshold 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_storage_usage_over_threshold 2025-10-27 11:14:36.220 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_percent_over_threshold, expression tenant500_mem_hold_percent > 95 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_percent_over_threshold 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_percent_over_threshold 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for vpc_connect_failed, expression vpc_network_health == 0 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='vpc_network_health', labels={}}], alarmTargets 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=vpc_connect_failed 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-vpc_connect_failed 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_fd_usage, expression observer_fd_usage > 65 and observer_fd_count > 0 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_fd_count', labels={}}, Metric{name='ob_host_observer_fd_usage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_fd_usage 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_fd_usage 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 97 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_sstable_percent_over_threshold, expression ob_disk_percent > 85 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_sstable_percent_over_threshold 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_sstable_percent_over_threshold 2025-10-27 11:14:36.221 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_absent, expression etcd_leader_count != 1 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_count', labels={}}], alarmTargets 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_absent 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_absent 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_mem_hold_over_threshold, expression tenant500_mem_hold > 100 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_mem_hold_gigabyte', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_mem_hold_over_threshold 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_mem_hold_over_threshold 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_exists_expired_xa_trans, expression xa_trans_max_duration_seconds > 1200 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_xa_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_exists_expired_xa_trans 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_exists_expired_xa_trans 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for agentd_process_stop, expression agentd_boot_time_delta_seconds > 10 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agentd_boot_time_delta_seconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agentd_process_stop 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agentd_process_stop 2025-10-27 11:14:36.222 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_merge, expression no_merge_time > 108000 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_merge_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_merge 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_merge 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_compaction, expression no_compaction_seconds > 108000 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_compaction_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_compaction 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_compaction 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_traffic_overload, expression eth_net_traffic_usage > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='host_eth_net_traffic_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_traffic_overload 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_traffic_overload 2025-10-27 11:14:36.223 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 97 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_mem_percent_over_threshold, expression memory_percent > 95 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_mem_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_mem_percent_over_threshold 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_mem_percent_over_threshold 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_connection_percent_over_threshold, expression ob_connection_percent > 80 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_connection_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_connection_percent_over_threshold 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_connection_percent_over_threshold 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for observer_process_stop, expression observer_boot_time_delta_seconds > 10 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='observer_boot_time_delta_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=observer_process_stop 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-observer_process_stop 2025-10-27 11:14:36.224 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_server_stopped, expression ob_server_stopped_duration_seconds > 0 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_stopped_duration_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_server_stopped 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_server_stopped 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 100 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_offset_too_large, expression ntp_offset_milliseconds > 50 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_offset_milliseconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_offset_too_large 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_offset_too_large 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_inode_usage, expression file_inode_usage > 80 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_file_inode_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_inode_usage 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_inode_usage 2025-10-27 11:14:36.225 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_delay_too_long, expression standby_tenant_delay_seconds >= 600 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_delay_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_delay_too_long 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_delay_too_long 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_expired_trans_exist, expression pending_trans_max_duration_seconds > 1200 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_pending_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_expired_trans_exist 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_expired_trans_exist 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_large_trans_exist, expression trans_max_log_size_mb > 102400 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_trans_max_log_size_mb', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_large_trans_exist 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_large_trans_exist 2025-10-27 11:14:36.226 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant500_storage_short_meta_mem_hold_high, expression tenant500_storage_short_meta_hold_percentage > 30 and tenant500_storage_short_meta_hold_gb > 0 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant500_storage_short_meta_hold_gb', labels={}}, Metric{name='tenant500_storage_short_meta_hold_percentage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant500_storage_short_meta_mem_hold_high 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant500_storage_short_meta_mem_hold_high 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_util_full, expression cpu_used_percent >= 98 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_used_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_util_full 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_util_full 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_switch_count', labels={}}], alarmTargets 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_switched 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_switched 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_ntp_service_not_exist, expression ntp_service_exist == 0 or ntp_service_available == 0 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_ntp_service_exist', labels={}}, Metric{name='host_ntp_service_available', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_ntp_service_not_exist 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_ntp_service_not_exist 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 2 2025-10-27 11:14:36.227 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for standby_tenant_sync_status_error, expression standby_tenant_restore_status_code == 1 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='standby_tenant_restore_status_code', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=standby_tenant_sync_status_error 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-standby_tenant_sync_status_error 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_task_timeout, expression ob_server_task_max_duration_seconds > 10800 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_task_max_duration', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_task_timeout 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_task_timeout 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_tsar_cpu_sys_abnormal, expression cpu_system > 40 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_cpu_system', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_tsar_cpu_sys_abnormal 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_tsar_cpu_sys_abnormal 2025-10-27 11:14:36.228 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_status_check_failed, expression ob_cluster_status_check == 0 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_status_check', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_status_check_failed 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_status_check_failed 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_invalid_disk_exists, expression ob_disk_invalid_count > 0 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_server_invalid_disk_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_invalid_disk_exists 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_invalid_disk_exists 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_master_switched, expression keepalived_release_master_count > 0 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_release_master_count', labels={}}], alarmTargets 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_master_switched 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_master_switched 2025-10-27 11:14:36.229 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_exists_inactive_server, expression oblb_inactive_server_count > 0 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_inactive_server_count', labels={}}], alarmTargets 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_exists_inactive_server 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_exists_inactive_server 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_tcp_retrans_percent_over_threshold, expression tcp_retrans_percent > 10 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_tcp_retrans_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_tcp_retrans_percent_over_threshold 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_tcp_retrans_percent_over_threshold 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_cpu_percent_over_threshold, expression cpu_percent > 90 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_cpu_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_cpu_percent_over_threshold 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_cpu_percent_over_threshold 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_process_dead, expression obdns_etcd_process_exists == 0 or obdns_etcd_connectable == 0 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_connectable', labels={}}, Metric{name='obdns_etcd_process_exists', labels={}}], alarmTargets 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_process_dead 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_process_dead 2025-10-27 11:14:36.230 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 2 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_load1_per_cpu_over_threshold, expression load1_per_cpu > 1.5 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_load1_per_cpu', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_load1_per_cpu_over_threshold 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_load1_per_cpu_over_threshold 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_meta_db_disconnected, expression ocp_meta_db_health == 0 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_meta_db_health', labels={}}], alarmTargets 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_meta_db_disconnected 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_meta_db_disconnected 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_host_cpu_usage_over_threshold, expression binlog_host_cpu_usage > 90 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_host_cpu_usage', labels={}}], alarmTargets C_binlogtest2-4/H_10.40.65.151 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_host_cpu_usage_over_threshold 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_host_cpu_usage_over_threshold 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_too_many_errors_occur, expression ocp_http_request_error_count >= 1 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_error_count', labels={}}], alarmTargets 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_too_many_errors_occur 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_too_many_errors_occur 2025-10-27 11:14:36.231 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_stop, expression obproxy_boot_time_delta_seconds > 10 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_boot_time_delta_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_stop 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_stop 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_compaction_error, expression compaction_is_error == 1 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_compaction_error_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_compaction_error 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_compaction_error 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_dump_error, expression binlog_dump_error_count > 10 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_dump_error_count', labels={}}], alarmTargets 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_dump_error 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_dump_error 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_disk_readonly, expression disk_readonly_flag == 1 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_disk_readonly_flag', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_disk_readonly 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_disk_readonly 2025-10-27 11:14:36.232 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_sync_delay_time_too_long, expression sync_delay_time > 600 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_sync_delay_time', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_sync_delay_time_too_long 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_sync_delay_time_too_long 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_http_request_timeout, expression ocp_http_request_duration_seconds >= 5 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_http_request_duration_seconds', labels={}}], alarmTargets 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_http_request_timeout 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_http_request_timeout 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for tenant_cpu_percent_over_threshold, expression ob_tenant_thread_percent > 80 2025-10-27 11:14:36.233 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=tenant_cpu_percent_over_threshold 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-tenant_cpu_percent_over_threshold 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_service_inactive, expression keepalived_process_exists == 0 or keepalived_up_status == 0 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_up_status', labels={}}, Metric{name='oblb_keepalived_process_exists', labels={}}], alarmTargets 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_service_inactive 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_service_inactive 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_etcd_leader_absent, expression etcd_leader_count != 1 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_etcd_leader_count', labels={}}], alarmTargets 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_etcd_leader_absent 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_etcd_leader_absent 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_data_disk_percent_over_threshold, expression ob_data_disk_usage_percent > 97 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_data_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_data_disk_percent_over_threshold 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_data_disk_percent_over_threshold 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_keepalived_become_master, expression keepalived_become_master_count > 0 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_keepalived_become_master_count', labels={}}], alarmTargets 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_keepalived_become_master 2025-10-27 11:14:36.234 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_keepalived_become_master 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for arbitration_group_auto_replace_failed, expression arbitration_group_auto_replace_task_status == 0 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='arbitration_group_auto_replace_task_status', labels={}}], alarmTargets 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=arbitration_group_auto_replace_failed 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-arbitration_group_auto_replace_failed 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_process_cpu_bind_core_is_invalid, expression host_process_cpu_bind_core_alarm_status == 1 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_process_cpu_bind_core_alarm_status', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_process_cpu_bind_core_is_invalid 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_process_cpu_bind_core_is_invalid 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for agent_process_count_abnormal, expression agent_process_count != 1 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='agent_process_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=agent_process_count_abnormal 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-agent_process_count_abnormal 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_disk_usage_high, expression ob_tenant_log_disk_usage > 85 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_disk_usage', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_disk_usage_high 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_disk_usage_high 2025-10-27 11:14:36.235 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration == 0 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration > 0 and ob_license_expired_second_duration < 2592000 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_license_expired, expression ob_license_expired_second_duration >= 2592000 and ob_license_expired_second_duration < 5184000 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_license_expired_second_duration', labels={}}], alarmTargets 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_license_expired 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_license_expired 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_service_name_is_invalid, expression ob_tenant_service_name_status == 1 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_service_name_status', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_service_name_is_invalid 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_service_name_is_invalid 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_instance_abnormal, expression binlog_instance_down > 0 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_instance_down', labels={}}], alarmTargets 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_instance_abnormal 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_instance_abnormal 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obdns_process_dead, expression obdns_process_exists == 0 or obdns_service_connectable == 0 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obdns_service_connectable', labels={}}, Metric{name='obdns_process_exists', labels={}}], alarmTargets 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obdns_process_dead 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obdns_process_dead 2025-10-27 11:14:36.236 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_merge_error, expression merge_error_flag == 1 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_merge_error_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_merge_error 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_merge_error 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_not_exist, expression observer_process_exists == 0 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_process_exists', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_not_exist 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_not_exist 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_open_fd_count_over_threshold, expression host_agent_open_fd_count > 1000 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_open_fd_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_open_fd_count_over_threshold 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_open_fd_count_over_threshold 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for backup_storage_capacity_timeout_or_interrupted, expression storage_capacity_monitor_error_code == 1002 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='storage_capacity_monitor_error_code', labels={}}], alarmTargets 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=backup_storage_capacity_timeout_or_interrupted 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-backup_storage_capacity_timeout_or_interrupted 2025-10-27 11:14:36.237 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_net_recv_percent_over_threshold, expression net_recv_percent > 80 and eth_net_bandwidth_mbps > 0 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_eth_net_bandwidth_mbps', labels={}}, Metric{name='ob_host_net_recv_percent', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_net_recv_percent_over_threshold 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_net_recv_percent_over_threshold 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_cannot_connect_to_leader, expression etcd_leader_connectable == 0 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_connectable', labels={}}], alarmTargets 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_cannot_connect_to_leader 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_cannot_connect_to_leader 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_collect_metric_failure_rate_high, expression collect_metric_failure_rate > 10 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ocp_collect_metric_failure_rate', labels={}}], alarmTargets 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_collect_metric_failure_rate_high 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_collect_metric_failure_rate_high 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_leader_switched, expression etcd_leader_switch_count > 0 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_leader_switch_count', labels={}}], alarmTargets 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_leader_switched 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_leader_switched 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxyd_process_dead, expression obproxyd_process_exists == 0 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxyd_process_exists', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxyd_process_dead 2025-10-27 11:14:36.238 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxyd_process_dead 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 97 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_install_disk_percent_over_threshold, expression ob_install_disk_usage_percent > 95 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_install_path_disk_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_install_disk_percent_over_threshold 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_install_disk_percent_over_threshold 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_no_frozen, expression no_frozen_time > 90000 2025-10-27 11:14:36.239 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_no_frozen_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_no_frozen 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_no_frozen 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_frozen_version_delta_over_threshold, expression frozen_versions > 1 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_frozen_version_delta', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_frozen_version_delta_over_threshold 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_frozen_version_delta_over_threshold 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for binlog_service_disk_usage_over_threshold, expression binlog_service_disk_usage > 90 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='binlog_service_disk_usage', labels={}}], alarmTargets 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=binlog_service_disk_usage_over_threshold 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-binlog_service_disk_usage_over_threshold 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_core_dump, expression obproxy_core_dump_seconds < 1200 and obproxy_core_dump_seconds > 0 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_core_dump_seconds', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_core_dump 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_core_dump 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_process_dead, expression obproxy_process_exists == 0 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_process_exists', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_process_dead 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_process_dead 2025-10-27 11:14:36.240 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for node_file_root_usage, expression root_disk_usage > 95 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_root_disk_usage', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=node_file_root_usage 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-node_file_root_usage 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_memory_peak, expression obproxy_rss_mem_used_percent > 90 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_rss_mem_used_percent', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_memory_peak 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_memory_peak 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_no_frozen, expression no_frozen_seconds > 90000 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_no_frozen_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_no_frozen 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_no_frozen 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_observer_core_dump, expression observer_core_dump_seconds < 1200 and observer_core_dump_seconds > 0 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_observer_core_dump_seconds', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_observer_core_dump 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_observer_core_dump 2025-10-27 11:14:36.241 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_partition_replica_absent, expression tenant_partition_replica_absent_count > 100 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_partition_replica_absent_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_partition_replica_absent 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_partition_replica_absent 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for monitor_exporter_unavaliable, expression monitor_exporter_avaliable == 0 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='monitor_exporter_avaliable', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=monitor_exporter_unavaliable 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-monitor_exporter_unavaliable 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for os_kernel_io_hang, expression io_util >= 99 and io_qusize >= 20 and io_read_write_time == 0 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_io_qusize', labels={}}, Metric{name='ob_host_io_util', labels={}}, Metric{name='ob_host_io_read_write_time', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=os_kernel_io_hang 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-os_kernel_io_hang 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for obproxy_client_connections_usage_over_threshold, expression client_connection_used_percent > 85 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='obproxy_client_connections_usage', labels={}}], alarmTargets C_obproxyTest-1/H_10.40.67.22 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=obproxy_client_connections_usage_over_threshold 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-obproxy_client_connections_usage_over_threshold 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_cpu_usage_over_threshold, expression ob_tenant_cpu_percent > 95 2025-10-27 11:14:36.242 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_real_cpu_percent', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_cpu_usage_over_threshold 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_cpu_usage_over_threshold 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_res_memory_over_threshold, expression host_agent_resident_memory_gb > 1.5 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_res_memory', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_res_memory_over_threshold 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_res_memory_over_threshold 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_unavailable, expression host_available == 0 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_available', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_unavailable 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_unavailable 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_task_timeout, expression ob_tenant_task_max_duration_seconds > 10800 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_task_max_duration', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_task_timeout 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_task_timeout 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_disk_readonly, expression disk_readonly_flag == 1 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_disk_readonly_flag', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_disk_readonly 2025-10-27 11:14:36.243 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_disk_readonly 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_long_trans_exist, expression long_trans_max_duration_seconds > 1200 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_long_trans_max_duration_seconds', labels={}}], alarmTargets C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY,C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_long_trans_exist 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_long_trans_exist 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_log_stream_degraded, expression ob_tenant_log_stream_degraded_count > 0 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_log_stream_degraded_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_log_stream_degraded 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_log_stream_degraded 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for host_agent_goroutine_count_over_threshold, expression host_agent_goroutine_count > 3000 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_agent_goroutine_count', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=host_agent_goroutine_count_over_threshold 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-host_agent_goroutine_count_over_threshold 2025-10-27 11:14:36.244 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cannot_connected, expression ob_connectable == 0 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_connectable', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cannot_connected 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cannot_connected 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for oblb_etcd_process_dead, expression etcd_process_exists == 0 or oblb_etcd_connectable == 0 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='oblb_etcd_connectable', labels={}}, Metric{name='oblb_etcd_process_exists', labels={}}], alarmTargets 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=oblb_etcd_process_dead 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-oblb_etcd_process_dead 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_inactive_server, expression inactive_server_count > 0 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_inactive_server_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_inactive_server 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_inactive_server 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_host_partition_count_over_threshold, expression partition_count > 30000 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_host_partition_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/H_10.40.65.146,H_10.40.65.168,H_10.40.67.22,C_myoceanbase2-1760510188-PRIMARY/H_10.40.66.107 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_host_partition_count_over_threshold 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_host_partition_count_over_threshold 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ocp_remote_server_time_diff_too_large, expression clock_diff_milliseconds > 50 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='host_clock_diff_milliseconds', labels={}}], alarmTargets H_10.40.65.146,H_10.40.65.151,H_10.40.65.168,H_10.40.66.107,H_10.40.67.22 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ocp_remote_server_time_diff_too_large 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ocp_remote_server_time_diff_too_large 2025-10-27 11:14:36.245 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 10 and request_queue_time > 1000 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_tenant_request_queue_over_threshold, expression ob_tenant_request_queue_size > 100 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_tenant_host_request_queue_time_us', labels={}}, Metric{name='ob_tenant_host_request_queue', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY/T_ocp_meta_PRIMARY,T_ocp_monitor_PRIMARY,T_sys_PRIMARY,T_test_PRIMARY,C_myoceanbase2-1760510188-PRIMARY/T_sys_PRIMARY,T_test2_PRIMARY 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_tenant_request_queue_over_threshold 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_tenant_request_queue_over_threshold 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : build for ob_cluster_exists_index_fail_table, expression index_fail_count > 0 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : metricsForQuery [Metric{name='ob_cluster_index_fail_count', labels={}}], alarmTargets C_myoceanbase-1760510187-PRIMARY,C_myoceanbase2-1760510188-PRIMARY 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.core.detect.AlarmRuleDetector : AlarmRuleDetector construct done, alarmType=ob_cluster_exists_index_fail_table 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : create done, key=0-ob_cluster_exists_index_fail_table 2025-10-27 11:14:36.246 INFO 2458 --- [ocp-schedules-24,0beb5cdde615f07c,41905f08631e2756] c.o.o.a.u.f.VersionalProductFactory : refresh done, configCount=134, updatedCount=134, failedToUpdateCount=0 2025-10-27 11:14:36.427 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:14:36.427 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:14:36.533 WARN 2458 --- [ocp-async-3,76c84ba0e2841df4,7f43bfbe03171670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:36.533 INFO 2458 --- [ocp-async-4,35c66901abbd1e6f,bb8dd268d6581622] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:36.539 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:36.540 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:14:36.552 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:36.552 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:14:36.612 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:36.613 INFO 2458 --- [ocp-schedules-27,aab0e54ffda64a2d,7d5c52dbd69b7fc1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:14:37.708 INFO 2458 --- [prometheus-cache-monitor1,75ebca09c059d242,34958fb1f05c088b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20314,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5096,"seriesIdKeySize":8772,"secondWriteQueue":0,"minuteWriteCacheSize":7869} 2025-10-27 11:14:37.708 INFO 2458 --- [prometheus-cache-monitor1,75ebca09c059d242,34958fb1f05c088b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:14:37.909 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:37.909 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:14:37.909 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:14:37.909 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:14:37.909 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:14:37.909 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:14:37.920 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:37.921 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:37.921 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:37.922 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:37.923 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:37.925 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:37.925 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:37.927 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:37.927 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:37.927 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:37.988 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:14:37.988 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:14:37.988 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:14:37.990 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:14:37.994 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:14:37.997 INFO 2458 --- [ocp-schedules-3,9a6abad44f4819a4,f11dfd4381b04b33] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:14:38.516 INFO 2458 --- [ocp-async-8,decf99d3cd64e3e4,a09fee4e3da66efe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:38.534 WARN 2458 --- [ocp-async-10,c0e5ba9b3075f3a0,1f772bf41f492bd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:38.534 INFO 2458 --- [ocp-async-6,bd3c427bb07cc7da,0687c66d69391c62] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:38.643 INFO 2458 --- [ocp-schedules-22,f0726225bc975439,6d5f94d22c0c3e59] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:14:38.650 INFO 2458 --- [ocp-schedules-22,f0726225bc975439,6d5f94d22c0c3e59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:38.651 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:14:38.651 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:14:38.651 INFO 2458 --- [ocp-schedules-22,f0726225bc975439,6d5f94d22c0c3e59] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:14:38.707 INFO 2458 --- [prometheus-minute-schedule-1,dd5a29b6bdd948ab,c3a4e8073fd888e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:38.715 INFO 2458 --- [prometheus-minute-schedule-1,dd5a29b6bdd948ab,c3a4e8073fd888e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:39.140 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:39.141 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:40.016 INFO 2458 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:14:40.020 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:14:40.056 INFO 2458 --- [Check obproxy1,56bcdee848f1a1b9,2781380cb83dce70] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:14:40.060 INFO 2458 --- [Check obproxy1,56bcdee848f1a1b9,2781380cb83dce70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:14:40.068 INFO 2458 --- [Collect obproxy1,e1a6d79c8a31a289,a2025a12a07266e5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:14:40.068 INFO 2458 --- [Collect obproxy1,e1a6d79c8a31a289,a2025a12a07266e5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:14:40.097 INFO 2458 --- [Check obproxy1,56bcdee848f1a1b9,2781380cb83dce70] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:14:40.100 INFO 2458 --- [Check obproxy1,56bcdee848f1a1b9,2781380cb83dce70] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:14:40.100 INFO 2458 --- [Check obproxy1,56bcdee848f1a1b9,2781380cb83dce70] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:14:40.101 INFO 2458 --- [Collect obproxy1,e1a6d79c8a31a289,a2025a12a07266e5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:14:40.104 INFO 2458 --- [Collect obproxy1,e1a6d79c8a31a289,a2025a12a07266e5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:14:40.106 INFO 2458 --- [Collect obproxy1,e1a6d79c8a31a289,a2025a12a07266e5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:14:40.534 WARN 2458 --- [ocp-async-9,636f29abd7a01412,b4ef99539d884ac1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:40.534 INFO 2458 --- [ocp-async-7,b89970868fa51aa7,4b9dcea95bfd2ae1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:40.589 INFO 2458 --- [ocp-schedules-12,91f357b78df8c632,6066fcfd2f090177] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:14:40.647 WARN 2458 --- [scanner-112154918-1,91f357b78df8c632,5fd7bacceb93628a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:14:40.680 INFO 2458 --- [alarm-detect-17,91f357b78df8c632,6848d22bd767b4ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:40.684 INFO 2458 --- [alarm-detect-17,91f357b78df8c632,6848d22bd767b4ce] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:14:40.703 INFO 2458 --- [alarm-detect-3,91f357b78df8c632,2d4bd74fc0a7ab8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:40.708 INFO 2458 --- [alarm-detect-2,91f357b78df8c632,36ffa8835968fff3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:40.708 INFO 2458 --- [alarm-detect-3,91f357b78df8c632,2d4bd74fc0a7ab8f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:14:40.711 INFO 2458 --- [alarm-detect-13,91f357b78df8c632,57796edb8268e2b9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534750 ~ 1761534810 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=203114.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203114.56, values={1761534780=203114.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:14:40.712 INFO 2458 --- [alarm-detect-2,91f357b78df8c632,36ffa8835968fff3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:14:40.712 INFO 2458 --- [alarm-detect-13,91f357b78df8c632,57796edb8268e2b9] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_compaction, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:14:40.712 INFO 2458 --- [alarm-detect-13,91f357b78df8c632,57796edb8268e2b9] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:14:40.801 INFO 2458 --- [alarm-detect-18,91f357b78df8c632,8ca7a3944c3b3772] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534750 ~ 1761534810 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=93.027)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.027, values={1761534780=93.027}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:14:40.801 INFO 2458 --- [alarm-detect-18,91f357b78df8c632,8ca7a3944c3b3772] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_log_disk_usage_high, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:14:40.802 INFO 2458 --- [alarm-detect-18,91f357b78df8c632,8ca7a3944c3b3772] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:14:40.829 INFO 2458 --- [alarm-detect-17,91f357b78df8c632,b3bd6c42b62b1723] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534750 ~ 1761534810 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534780, value=203115.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203115.912, values={1761534780=203115.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:14:40.829 INFO 2458 --- [alarm-detect-17,91f357b78df8c632,b3bd6c42b62b1723] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_frozen, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:14:40.830 INFO 2458 --- [alarm-detect-17,91f357b78df8c632,b3bd6c42b62b1723] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:14:40.830 INFO 2458 --- [alarm-detect-4,91f357b78df8c632,f94350131ce7342a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:40.835 INFO 2458 --- [alarm-detect-4,91f357b78df8c632,f94350131ce7342a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:14:40.857 INFO 2458 --- [alarm-detect-19,91f357b78df8c632,b7e977768331ad61] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534840 ~ 1761534870 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534870, value=62.353), ValueData(timestamp=1761534855, value=70.091)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534870, value=17409.0), ValueData(timestamp=1761534855, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534870=17409.0, 1761534855=17409.0}, valueCount=2, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:14:40.861 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,30d21e5c924b833a,483a75af2ba56ad5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=30d21e5c924b833a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:14:40.861 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,30d21e5c924b833a,483a75af2ba56ad5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=30d21e5c924b833a, duration=1 ms] 2025-10-27 11:14:40.880 INFO 2458 --- [alarm-detect-19,91f357b78df8c632,b7e977768331ad61] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:14:40.909 INFO 2458 --- [ocp-schedules-27,9d04138a110249a9,1abb58efa92b80da] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:14:40.910 INFO 2458 --- [ocp-schedules-27,9d04138a110249a9,1abb58efa92b80da] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:14:41.707 INFO 2458 --- [prometheus-minute-schedule-1,17bcc0ad2a597c49,910e1e53ac0ed6ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:41.713 INFO 2458 --- [prometheus-minute-schedule-1,17bcc0ad2a597c49,910e1e53ac0ed6ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:41.714258+08:00, lastProcessTime=2025-10-27T11:14:31.718902+08:00 2025-10-27 11:14:41.938 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:41.714258+08:00 2025-10-27 11:14:42.141 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:42.141 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:42.534 WARN 2458 --- [ocp-async-1,10d4f39ebe35de10,1ab1cc4a81631726] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:42.534 INFO 2458 --- [ocp-async-1,6d6c94d116e9fee3,2b4870cd653c7235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:44.055 INFO 2458 --- [ocp-schedules-15,e552f5a2f853822c,928c6d40084ef334] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:14:44.102 INFO 2458 --- [ocp-schedules-13,daedb082fb89b8cf,cb77a6562b5b5a90] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:44.105 INFO 2458 --- [ocp-schedules-13,daedb082fb89b8cf,cb77a6562b5b5a90] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:44.534 WARN 2458 --- [ocp-async-10,d1b8845b10719102,a9cf56a612cf08b2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:44.534 INFO 2458 --- [ocp-async-8,100ca094c8687ac6,ba29bcd36104792d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:44.707 INFO 2458 --- [prometheus-minute-schedule-1,c5cf5e1fce4ec2f4,ea8cfe07fe57272d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:44.714 INFO 2458 --- [prometheus-minute-schedule-1,c5cf5e1fce4ec2f4,ea8cfe07fe57272d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:45.102 INFO 2458 --- [prometheus-second-schedule-1,b04525507a678a71,672f9cc5b26f1e02] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:14:45.142 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:45.142 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:46.011 INFO 2458 --- [ocp-schedules-12,4513586564fca4fd,16431afd362153a2] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:14:46.011 INFO 2458 --- [ocp-schedules-12,4513586564fca4fd,16431afd362153a2] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:14:46.067 INFO 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:14:46.068 INFO 2458 --- [pool-878-thread-2,4513586564fca4fd,68cbe3423d30c85a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:14:46.135 INFO 2458 --- [pool-878-thread-2,4513586564fca4fd,68cbe3423d30c85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:46.138 INFO 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:46.139 INFO 2458 --- [pool-878-thread-2,4513586564fca4fd,68cbe3423d30c85a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534585000000, 1761534886000000] 2025-10-27 11:14:46.143 INFO 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534585000000, 1761534886000000] 2025-10-27 11:14:46.183 WARN 2458 --- [pool-878-thread-2,4513586564fca4fd,68cbe3423d30c85a] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:14:46.184 INFO 2458 --- [ocp-schedules-23,6ecf3b05e4774878,7043815afb5b5871] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:46.196 INFO 2458 --- [pool-878-thread-2,4513586564fca4fd,68cbe3423d30c85a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:14:46.275 WARN 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:14:46.275 WARN 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:14:46.275 WARN 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:14:46.283 INFO 2458 --- [pool-878-thread-1,4513586564fca4fd,7e27d585cc72fb5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:14:46.284 INFO 2458 --- [ocp-schedules-12,4513586564fca4fd,16431afd362153a2] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:14:46.534 INFO 2458 --- [ocp-async-6,6daeb81c0d22258a,9ff4118d41bd350e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:46.534 WARN 2458 --- [ocp-async-9,fea78d7c236b2a18,594d3e83a15da74a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:46.969 INFO 2458 --- [ocp-schedules-19,d04b5dece6b19be9,c0a875bbfd4cff56] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:14:46.976 INFO 2458 --- [ocp-schedules-19,d04b5dece6b19be9,c0a875bbfd4cff56] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:46.976 INFO 2458 --- [ocp-schedules-19,d04b5dece6b19be9,c0a875bbfd4cff56] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:14:47.443 INFO 2458 --- [ocp-schedules-28,a3475a51a5192ad2,82bc0732637ce4e3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:14:47.443 INFO 2458 --- [ocp-schedules-28,a3475a51a5192ad2,82bc0732637ce4e3] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:14:47.450 INFO 2458 --- [ocp-schedules-28,a3475a51a5192ad2,82bc0732637ce4e3] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:14:47.451 INFO 2458 --- [ocp-schedules-28,a3475a51a5192ad2,82bc0732637ce4e3] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:14:47.707 INFO 2458 --- [prometheus-minute-schedule-1,a31d69813a3ddda7,cbe9b5b7141c834f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:47.714 INFO 2458 --- [prometheus-minute-schedule-1,a31d69813a3ddda7,cbe9b5b7141c834f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:48.142 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:48.144 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:48.406 INFO 2458 --- [ocp-schedules-4,17b4fe985eebe906,ddf43ac17c3e4f13] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:14:48.406+08:00, clusters=[1, 2] 2025-10-27 11:14:48.421 INFO 2458 --- [ocp-schedules-4,17b4fe985eebe906,ddf43ac17c3e4f13] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:14:48.424 INFO 2458 --- [ocp-schedules-4,17b4fe985eebe906,ddf43ac17c3e4f13] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:14:48.516 INFO 2458 --- [ocp-async-6,d949142ca64bcaec,fec0e7ba75912c3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:48.534 INFO 2458 --- [ocp-async-4,a27b2019cba8f9f2,e5d28946d4661e3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:48.534 WARN 2458 --- [ocp-async-1,0c602a252e9169e3,1adf5d12c224d2bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:50.534 WARN 2458 --- [ocp-async-2,007b643ba9e1b164,01409264e3e227f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:50.534 INFO 2458 --- [ocp-async-2,ce0ffe768b18b5e9,439e93c8a288d677] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:50.707 INFO 2458 --- [prometheus-minute-schedule-1,962ecb83bf46126d,c60635d5abf6de99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:50.714 INFO 2458 --- [prometheus-minute-schedule-1,962ecb83bf46126d,c60635d5abf6de99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:50.880 INFO 2458 --- [ocp-schedules-7,670de9ae4e79d2e8,b67093cf2239d2ba] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:14:50.909 WARN 2458 --- [scanner-112155198-1,670de9ae4e79d2e8,724937d692c0ce7b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:14:51.144 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:51.144 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:51.408 INFO 2458 --- [ocp-schedules-18,4bde11a4bed2f635,b4cb067b79c34a16] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:51.408 INFO 2458 --- [ocp-schedules-24,2d7e926a43e0eb32,ffae964d4fe3d572] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:51.408 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:14:51.408 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:14:51.409 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:14:51.409 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:14:51.410 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:14:51.424 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:51.425 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:51.425 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:51.426 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:51.426 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:51.426 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:51.426 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:51.429 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:51.429 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:51.430 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:51.431 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:51.432 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:51.433 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:51.434 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:51.436 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:51.439 INFO 2458 --- [ocp-schedules-26,5cd41cd11cdd4cc3,1b31f16bc8da6d40] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:14:51.439 INFO 2458 --- [host-check-clockdiff1,5cd41cd11cdd4cc3,5b970212a59960e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:14:51.440 INFO 2458 --- [host-check-clockdiff3,5cd41cd11cdd4cc3,421e804cf5619a66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:14:51.440 INFO 2458 --- [host-check-clockdiff4,5cd41cd11cdd4cc3,ea5a9fc6067998a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:14:51.440 INFO 2458 --- [host-check-clockdiff5,5cd41cd11cdd4cc3,dfc1acc33732249a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:14:51.440 INFO 2458 --- [host-check-clockdiff2,5cd41cd11cdd4cc3,48388091383f9da3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:14:51.442 INFO 2458 --- [ocp-schedules-30,aba69b3654c114b3,7c80a04dbb9076c2] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:14:51.466 INFO 2458 --- [host-check-clockdiff2,5cd41cd11cdd4cc3,48388091383f9da3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534891 -1 -1 , error= 2025-10-27 11:14:51.473 INFO 2458 --- [host-check-clockdiff5,5cd41cd11cdd4cc3,dfc1acc33732249a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534891 0 0 , error= 2025-10-27 11:14:51.473 INFO 2458 --- [host-check-clockdiff3,5cd41cd11cdd4cc3,421e804cf5619a66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534891 0 0 , error= 2025-10-27 11:14:51.499 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:14:51.500 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:14:51.501 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:14:51.500788919+08:00 2025-10-27 11:14:51.501 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:14:51.501+08:00 2025-10-27 11:14:51.502 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:14:51.504 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:14:51.504 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:14:51.504 INFO 2458 --- [host-check-agent-status3,2d7e926a43e0eb32,540a07b1e62c0dfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:14:51.504 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:14:51.506 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:14:51.506 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:14:51.506 INFO 2458 --- [host-check-agent-status2,2d7e926a43e0eb32,a6389c392941c314] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:14:51.507 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:14:51.507 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:14:51.507 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:14:51.507 INFO 2458 --- [host-check-agent-status1,2d7e926a43e0eb32,5cc00302c187a82c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:14:51.508 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:14:51.510 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:14:51.509737200+08:00 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status5,2d7e926a43e0eb32,a10c3be237bea07d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:14:51.511+08:00 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:14:51.511 INFO 2458 --- [host-check-agent-status4,2d7e926a43e0eb32,e0077417b97393b0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:14:51.513 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:14:51.512480439+08:00 2025-10-27 11:14:51.513 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:14:51.513 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:14:51.513+08:00 2025-10-27 11:14:51.514 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:14:51.513239560+08:00 2025-10-27 11:14:51.514 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:14:51.514+08:00 2025-10-27 11:14:51.514 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:14:51.501+08:00 and status AVAILABLE 2025-10-27 11:14:51.518 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:14:51.511+08:00 and status AVAILABLE 2025-10-27 11:14:51.520 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:14:51.520 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:14:51.513+08:00 and status AVAILABLE 2025-10-27 11:14:51.521 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:14:51.522 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:14:51.522028738+08:00 2025-10-27 11:14:51.522 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:14:51.522+08:00 2025-10-27 11:14:51.522 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:14:51.514+08:00 and status AVAILABLE 2025-10-27 11:14:51.525 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:14:51.527 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:14:51.527 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:14:51.529 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:14:51.522+08:00 and status AVAILABLE 2025-10-27 11:14:51.533 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:14:51.539 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:51.542 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:51.544 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:14:51.545 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:14:51.545 INFO 2458 --- [ocp-schedules-27,9709706c06cd911f,2557ad59d0363e36] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:14:51.546 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:14:51.546 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:14:51.548 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:14:51.549 INFO 2458 --- [ocp-schedules-25,23a292bac29338bb,f660a7b5e37a9a4f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:14:51.549 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:14:51.550 INFO 2458 --- [ocp-schedules-27,9709706c06cd911f,2557ad59d0363e36] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:51.553 INFO 2458 --- [ocp-schedules-25,23a292bac29338bb,f660a7b5e37a9a4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:51.554 INFO 2458 --- [ocp-schedules-13,9b5132c79c9513c3,0221daff9c33446c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:14:51.554 INFO 2458 --- [ocp-schedules-25,23a292bac29338bb,f660a7b5e37a9a4f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:14:51.554 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:51.557 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:51.559 INFO 2458 --- [ocp-schedules-13,9b5132c79c9513c3,0221daff9c33446c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:14:51.566 INFO 2458 --- [ocp-schedules-8,9ae15df812ba3dda,6a021810a30f1c9c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:14:51.568 INFO 2458 --- [ocp-schedules-1,07807f3cee3d44ca,f6b8666b65b07cc5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:14:51.569 INFO 2458 --- [ocp-schedules-9,947b111e69557e29,071ff640542584e0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:14:51.579 INFO 2458 --- [ocp-schedules-1,07807f3cee3d44ca,f6b8666b65b07cc5] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:14:51.582 INFO 2458 --- [ocp-schedules-9,947b111e69557e29,071ff640542584e0] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:14:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:14:51.604 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:51.606 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:51.611 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:51.612 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:51.615 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.616 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:14:51.623 INFO 2458 --- [ocp-schedules-27,9709706c06cd911f,2557ad59d0363e36] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:14:51.625 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:14:51.626 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:14:51.626 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:14:51.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:51.633 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:14:51.636 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:14:51.647 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:14:51.660 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:14:51.661 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:14:51.663 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:14:51.664 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:14:51.675 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.676 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:14:51.676 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:51.686 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:51.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:14:51.702 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:51.706 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:51.711 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:51.712 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:14:51.712448+08:00, lastProcessTime=2025-10-27T11:14:41.714258+08:00 2025-10-27 11:14:51.719 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:14:51.729 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:14:51.732 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:14:51.734 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:14:51.737 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:14:51.743 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:14:51.751 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:51.757 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:51.757 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.757 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.758 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:14:51.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:14:51.760 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:14:51.761 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:51.762 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:14:51.763 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:14:51.768 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:14:51.783 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:14:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:51.800 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:51.806 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:51.810 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:51.820 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:51.822 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:14:51.822 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:14:51.823 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:14:51.832 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:14:51.833 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:51.835 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:14:51.852 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:14:51.852 INFO 2458 --- [host-check-host-status2,4bde11a4bed2f635,cb9862cba184fad6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:14:51.859 INFO 2458 --- [host-check-host-status5,4bde11a4bed2f635,fc620c0f3f226e20] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:14:51.861 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:51.862 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.862 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:51.864 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:14:51.870 INFO 2458 --- [host-check-host-status4,4bde11a4bed2f635,b9464b23431c26f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:14:51.871 INFO 2458 --- [host-check-host-status3,4bde11a4bed2f635,12264ab41a855470] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:14:51.871 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:51.872 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:14:51.874 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:51.874 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:14:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:14:51.879 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:51.884 INFO 2458 --- [host-check-host-status1,4bde11a4bed2f635,4e526749597253ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:14:51.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:14:51.891 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:14:51.956 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:51.956 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:14:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:52.004 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:14:51.712448+08:00 2025-10-27 11:14:52.024 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:14:52.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.025 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:52.025 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:14:52.025 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:52.030 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.031 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:14:52.047 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:14:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:14:52.071 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:52.075 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:14:52.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:52.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:14:52.097 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:14:52.097 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:14:52.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:14:52.167 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:52.167 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.168 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:14:52.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:14:52.180 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:14:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:14:52.192 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:14:52.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:14:52.304 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:14:52.305 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.305 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:14:52.307 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.308 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:14:52.344 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:14:52.381 INFO 2458 --- [host-check-clockdiff1,5cd41cd11cdd4cc3,5b970212a59960e9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534892 0 0 , error= 2025-10-27 11:14:52.385 INFO 2458 --- [host-check-clockdiff4,5cd41cd11cdd4cc3,ea5a9fc6067998a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534892 0 0 , error= 2025-10-27 11:14:52.387 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:14:52.388 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:14:52.389 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:14:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:14:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:14:52.431 INFO 2458 --- [ocp-schedules-13,9b5132c79c9513c3,0221daff9c33446c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:14:52.535 WARN 2458 --- [ocp-async-9,adf0cb89d54cf639,3f5b3cf556ca6e86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:52.535 INFO 2458 --- [ocp-async-7,0b462418a28393f4,193399b73b4275be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:14:53.707 INFO 2458 --- [prometheus-minute-schedule-1,eaa9651680994a7d,4526dc1c7410341c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:53.715 INFO 2458 --- [prometheus-minute-schedule-1,eaa9651680994a7d,4526dc1c7410341c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:54.115 INFO 2458 --- [ocp-schedules-20,de46cb38de40c7eb,9f942c16ca8adf1d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:14:54.118 INFO 2458 --- [ocp-schedules-20,de46cb38de40c7eb,9f942c16ca8adf1d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:14:54.145 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:54.146 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:54.535 WARN 2458 --- [ocp-async-7,5285f9d5b8de34d3,5d67f3a3768b4ca9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:54.535 INFO 2458 --- [ocp-async-6,4f3818fa3a37a55c,faee249e5019e683] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:54.750 INFO 2458 --- [ocp-schedules-17,73da9ecd76dcfbbf,428b0fa9ff9c1e77] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:14:56.200 INFO 2458 --- [ocp-schedules-2,2345f8ef17e45902,27b711e808cc7b27] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:14:56.535 WARN 2458 --- [ocp-async-6,30a528708d501a80,b52be0c426e5823c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:56.535 INFO 2458 --- [ocp-async-10,9109fb3bd18cccf4,177cf5e039c51f01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:56.707 INFO 2458 --- [prometheus-minute-schedule-1,53797db38589f409,253ea870d4b5805a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:56.714 INFO 2458 --- [prometheus-minute-schedule-1,53797db38589f409,253ea870d4b5805a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:14:57.146 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:14:57.147 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:14:58.516 INFO 2458 --- [ocp-async-10,993c710e2d6aa9e4,ed7a0ab4fcb4ba80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:14:58.535 INFO 2458 --- [ocp-async-1,8feefea292ce2700,4574954689eb0d32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:14:58.535 WARN 2458 --- [ocp-async-9,aff25ae93c61b9e7,458698dd0740bdbb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:14:59.277 INFO 2458 --- [ocp-schedules-8,629ec22abd4e3bc1,4d4f4798f97b7bdd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:14:59.707 INFO 2458 --- [prometheus-minute-schedule-1,40250e2d519b547e,079321f472450982] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534900, interval=60 2025-10-27 11:14:59.715 INFO 2458 --- [prometheus-minute-schedule-1,40250e2d519b547e,079321f472450982] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:00.147 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:00.148 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:00.535 INFO 2458 --- [ocp-async-1,94ff34cc25716457,23032980d985eb76] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:00.535 WARN 2458 --- [ocp-async-4,dddfe2ff570a2e0d,90060a2ba8780f75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:00.967 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:10:01.0, nextRun=2025-10-27 11:15:00.0, currentDate=Mon Oct 27 11:15:00 CST 2025 2025-10-27 11:15:00.967 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:10:01.0, nextRunTime=2025-10-27 11:15:00.0} 2025-10-27 11:15:00.981 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:15:00.981 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:15:00.981+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:15:00.994 INFO 2458 --- [ocp-schedules-24,207b07dcb5063de0,b93408f8f638503a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:01.008 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:10:01.0, nextRun=2025-10-27 11:15:00.0, currentDate=Mon Oct 27 11:15:01 CST 2025 2025-10-27 11:15:01.008 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:10:01.0, nextRunTime=2025-10-27 11:15:00.0} 2025-10-27 11:15:01.057 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,9d570fba517d5b54] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9158, operation:EXECUTE, state: RUNNING 2025-10-27 11:15:01.069 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:15:01.088 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:15:01.089 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9158, context=Context{parallelIdx=-1, stringMap={task_instance_id=9100, task_operation=execute, latest_execution_start_time=2025-10-27T11:15:01.051+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9158}, listMap={}}, executor=10.40.65.146 2025-10-27 11:15:01.108 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:15:01.115 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:15:01.115 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,13046f0d7d80a8da] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:15:01.115+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:15:01.119 WARN 2458 --- [scanner-112155386-1,207b07dcb5063de0,0a467ed1541ff80a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:01.177 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:01.186 INFO 2458 --- [alarm-detect-11,207b07dcb5063de0,e62fcb167567b94c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:01.192 INFO 2458 --- [alarm-detect-11,207b07dcb5063de0,e62fcb167567b94c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:01.251 INFO 2458 --- [alarm-detect-18,207b07dcb5063de0,524ff7d0e46789b9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:01.253 INFO 2458 --- [alarm-detect-6,207b07dcb5063de0,090db443da90ec84] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:01.257 INFO 2458 --- [alarm-detect-18,207b07dcb5063de0,524ff7d0e46789b9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:01.259 INFO 2458 --- [alarm-detect-6,207b07dcb5063de0,090db443da90ec84] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:01.456 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:01.504 INFO 2458 --- [alarm-detect-2,207b07dcb5063de0,2390f5342c6d8c58] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:01.510 INFO 2458 --- [alarm-detect-2,207b07dcb5063de0,2390f5342c6d8c58] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:01.638 INFO 2458 --- [alarm-detect-15,207b07dcb5063de0,042ef5416745a723] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534870 ~ 1761534891 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534885, value=53.115)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534885, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534885=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:15:01.661 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:01.715337+08:00, lastProcessTime=2025-10-27T11:14:51.712448+08:00 2025-10-27 11:15:01.744 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:01.904 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:01.986 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:01.715337+08:00 2025-10-27 11:15:01.998 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:02.075 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,2a617651f79e7151] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9159, operation:EXECUTE, state: RUNNING 2025-10-27 11:15:02.109 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:15:02.113 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9159, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9101, task_operation=execute, latest_execution_start_time=2025-10-27T11:15:02.072+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9159}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:15:02.114 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:15:02.138 INFO 2458 --- [ocp-schedules-11,a2ba01e2434fa365,420e77869bda81be] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:15:02.148 INFO 2458 --- [ocp-schedules-11,a2ba01e2434fa365,420e77869bda81be] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:02.220 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:02.268 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:15:02.268 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.273 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:15:02.275 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:15:02.281 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:15:02.286 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.287 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.317 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.317 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.322 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:15:02.331 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:15:02.331 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.334 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:15:02.344 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:15:02.344 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:15:02.351 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.352 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.353 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.355 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.356 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.357 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.358 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.413 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.427 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.427 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.428 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.428 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.439 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:15:02.440 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.444 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:15:02.460 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.461 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:15:02.472 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.474 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.476 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.503 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.504 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.505 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.505 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.505 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.507 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.507 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.508 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.509 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.510 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.511 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.512 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.513 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.514 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.514 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:15:02.522 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:15:02.535 INFO 2458 --- [ocp-async-1,94b1461f4ffbfa19,4c76586dac99a5a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:02.535 WARN 2458 --- [ocp-async-6,5b72110d483dea58,dda6f7e7e980bc91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:02.537 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.539 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:15:02.542 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.543 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:15:02.550 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.560 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:15:02.562 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:15:02.562 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:15:02.576 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.577 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:15:02.619 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.619 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:15:02.623 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.625 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:15:02.639 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:02.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:15:02.643 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.652 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:02.652 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:15:02.652 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:02.652 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:15:02.657 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:15:02.657 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:15:02.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:15:02.707 INFO 2458 --- [prometheus-minute-schedule-1,346faea1cb618a15,76414a24ad745ad4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:02.708 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:15:02.712 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.perf.sql.internal.SqlParseService : parse 16 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:15:02.718 INFO 2458 --- [prometheus-minute-schedule-1,346faea1cb618a15,76414a24ad745ad4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:02.731 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.735 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:15:02.772 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.796 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.797 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.798 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.799 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.800 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.801 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.802 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.803 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.804 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.805 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.805 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.805 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:15:02.806 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.809 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.810 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.811 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.812 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.814 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.815 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.817 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.818 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:15:02.820 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.821 INFO 2458 --- [ocp-schedules-21,cc76d7e9da4f0912,d3b907ac8798bbc0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:15:02.821 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:15:02.822 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:15:02.822 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:15:02.822 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.824 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:15:02.826 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:15:02.829 INFO 2458 --- [ocp-schedules-21,cc76d7e9da4f0912,d3b907ac8798bbc0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:02.830 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:15:02.840 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,ba5b7069ae95a039] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9158, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:15:02.889 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.889 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:15:02.929 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.930 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:15:02.953 INFO 2458 --- [ocp-schedules-21,cc76d7e9da4f0912,d3b907ac8798bbc0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:15:02.960 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:02.961 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:02.961 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:15:03.008 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:15:03.012 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:15:03.012 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:15:03.013 INFO 2458 --- [ocp-schedules-11,a2ba01e2434fa365,420e77869bda81be] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:15:03.149 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:03.149 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:03.322 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,2880c5864c5ed196] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9159, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:15:03.497 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9100, status:SUCCESS 2025-10-27 11:15:03.505 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9100, status:SUCCESS 2025-10-27 11:15:03.515 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9101, status:SUCCESS 2025-10-27 11:15:03.515 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9101, status:SUCCESS 2025-10-27 11:15:04.128 INFO 2458 --- [ocp-schedules-30,49cb374b154315a3,294501e3b1a7abdb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:04.131 INFO 2458 --- [ocp-schedules-30,49cb374b154315a3,294501e3b1a7abdb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:04.535 INFO 2458 --- [ocp-async-10,577cf16488a90d3b,b250c2dd7c692c37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:04.536 WARN 2458 --- [ocp-async-3,14c03e80d0e3db59,9940ee2d8bfa7c9a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:05.707 INFO 2458 --- [prometheus-minute-schedule-1,fd073386d94b9959,f733e075b32d136e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:05.715 INFO 2458 --- [prometheus-minute-schedule-1,fd073386d94b9959,f733e075b32d136e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:06.150 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:06.151 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:06.207 INFO 2458 --- [ocp-schedules-16,ae921aad46a82e7b,d1cc2b066de5a02e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:06.536 INFO 2458 --- [ocp-async-4,94a3ab584f2de3ca,b0a919a2aa3bb7b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:06.536 WARN 2458 --- [ocp-async-8,2b8b32c0d3f1b2a9,885754d8a3cca374] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:06.980 INFO 2458 --- [ocp-schedules-26,4a57405e4d8af23d,2fc7e0bb53b6e505] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:15:06.988 INFO 2458 --- [ocp-schedules-26,4a57405e4d8af23d,2fc7e0bb53b6e505] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:06.988 INFO 2458 --- [ocp-schedules-26,4a57405e4d8af23d,2fc7e0bb53b6e505] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:08.516 INFO 2458 --- [ocp-async-6,b05406d0efcda7e0,58d5f86c3a1086a8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:08.536 INFO 2458 --- [ocp-async-5,7aa38692fc13ec1b,3a4c906848bb2767] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:08.536 WARN 2458 --- [ocp-async-1,0bfa3f8fedd180f1,e963047c186452f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:08.707 INFO 2458 --- [prometheus-minute-schedule-1,91cdcef2bfcfc30b,7a8817d48aa9e93a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:08.715 INFO 2458 --- [prometheus-minute-schedule-1,91cdcef2bfcfc30b,7a8817d48aa9e93a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:09.151 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:09.152 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:10.013 INFO 2458 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:15:10.013 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:15:10.051 INFO 2458 --- [Check obproxy1,3fd071cdf04e2e63,7d1e2c4d2baa189d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:15:10.051 INFO 2458 --- [Check obproxy1,3fd071cdf04e2e63,7d1e2c4d2baa189d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:15:10.053 INFO 2458 --- [Collect obproxy1,5aac9c6a6db69d90,99a2b2ecab6f5a32] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:15:10.053 INFO 2458 --- [Collect obproxy1,5aac9c6a6db69d90,99a2b2ecab6f5a32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:15:10.086 INFO 2458 --- [Check obproxy1,3fd071cdf04e2e63,7d1e2c4d2baa189d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:15:10.088 INFO 2458 --- [Collect obproxy1,5aac9c6a6db69d90,99a2b2ecab6f5a32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:15:10.090 INFO 2458 --- [Check obproxy1,3fd071cdf04e2e63,7d1e2c4d2baa189d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:15:10.090 INFO 2458 --- [Check obproxy1,3fd071cdf04e2e63,7d1e2c4d2baa189d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:15:10.092 INFO 2458 --- [Collect obproxy1,5aac9c6a6db69d90,99a2b2ecab6f5a32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:15:10.095 INFO 2458 --- [Collect obproxy1,5aac9c6a6db69d90,99a2b2ecab6f5a32] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:15:10.184 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:15:10.536 INFO 2458 --- [ocp-async-3,cf1f8589cf6610dd,65197544572e0396] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:10.536 WARN 2458 --- [ocp-async-7,d4e200d17d0f1ed1,519952162d479c18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:10.730 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534840, nodeCount=496, affectRows=496 2025-10-27 11:15:10.764 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534840, nodeCount=227, affectRows=227 2025-10-27 11:15:10.780 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534840, nodeCount=34, affectRows=34 2025-10-27 11:15:10.798 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534840, nodeCount=49, affectRows=49 2025-10-27 11:15:10.807 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534840, nodeCount=15, affectRows=15 2025-10-27 11:15:11.645 INFO 2458 --- [ocp-schedules-21,127169259319e018,a58860a185f44622] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:11.672 WARN 2458 --- [scanner-112157348-1,127169259319e018,29bec4a0d313d4a9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:11.707 INFO 2458 --- [prometheus-minute-schedule-1,eb9047a6a530331e,c84ac62eecdc209f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:11.720 INFO 2458 --- [prometheus-minute-schedule-1,eb9047a6a530331e,c84ac62eecdc209f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:11.727 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:11.726101+08:00, lastProcessTime=2025-10-27T11:15:01.715337+08:00 2025-10-27 11:15:11.840 INFO 2458 --- [alarm-detect-1,127169259319e018,ab36bfc21974632d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534885 ~ 1761534901 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534900, value=44.274)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534900, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534900=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:15:11.978 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:11.726101+08:00 2025-10-27 11:15:12.152 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:12.153 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:12.536 INFO 2458 --- [ocp-async-8,f09ef07517a0f044,03a7fb9be6b3e8e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:12.536 WARN 2458 --- [ocp-async-2,4d0d973a9c7cdc48,77b6404cde7485bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:13.271 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=1000, affectRows=1000 2025-10-27 11:15:13.275 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=994, validCount=992 2025-10-27 11:15:13.366 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=1000, affectRows=1000 2025-10-27 11:15:13.370 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=999, validCount=995 2025-10-27 11:15:13.385 ERROR 2458 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538020) Duplicate entry '835370602979598336-1761534780' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538020) Duplicate entry '835370602979598336-1761534780' for key 'PRIMARY' 2025-10-27 11:15:14.150 INFO 2458 --- [ocp-schedules-24,de9a2f98dfea46e6,c48bc616c6465a6e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:14.153 INFO 2458 --- [ocp-schedules-24,de9a2f98dfea46e6,c48bc616c6465a6e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:14.536 WARN 2458 --- [ocp-async-1,6549fa8d01c8068f,2fbe1d0c132e397d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:14.536 INFO 2458 --- [ocp-async-6,a4c1c7d3de197127,00a0a1718e26eddc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:14.707 INFO 2458 --- [prometheus-minute-schedule-1,84517eb539719cf3,4cdf408be362cef8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:14.714 INFO 2458 --- [prometheus-minute-schedule-1,84517eb539719cf3,4cdf408be362cef8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:14.845 INFO 2458 --- [ocp-schedules-4,42b68d07ae567246,b76c8017859f6b95] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:15:14.845+08:00, clusters=[1, 2] 2025-10-27 11:15:14.847 INFO 2458 --- [ocp-schedules-4,42b68d07ae567246,b76c8017859f6b95] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:15:14.847 INFO 2458 --- [ocp-schedules-4,42b68d07ae567246,b76c8017859f6b95] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:15:15.153 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:15.154 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:15.406 INFO 2458 --- [ocp-schedules-7,056476d45ea0fc5c,f52db81806175865] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:15:15.406 INFO 2458 --- [ocp-schedules-7,056476d45ea0fc5c,f52db81806175865] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:15:15.445 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=98, affectRows=98 2025-10-27 11:15:15.445 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=97, validCount=97 2025-10-27 11:15:15.471 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=1000, affectRows=1000 2025-10-27 11:15:15.475 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=1000, validCount=996 2025-10-27 11:15:15.477 INFO 2458 --- [Collect all tenants major compactions1,056476d45ea0fc5c,d95221b77d8dccf3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:15:15.478 INFO 2458 --- [Collect all tenants major compactions2,056476d45ea0fc5c,75a94ac020cc4297] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:15:15.478 INFO 2458 --- [Collect all tenants major compactions4,056476d45ea0fc5c,2948a3e449a06aac] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:15:15.478 INFO 2458 --- [Collect all tenants major compactions3,056476d45ea0fc5c,f21efca1ca4b42e1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:15:15.478 INFO 2458 --- [Collect all tenants major compactions5,056476d45ea0fc5c,099706fd485e28ad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:15:15.479 INFO 2458 --- [Collect all tenants major compactions6,056476d45ea0fc5c,fb2d44fe1ed2c78a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:15:15.508 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=1000, affectRows=1000 2025-10-27 11:15:15.516 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=996, validCount=992 2025-10-27 11:15:15.521 INFO 2458 --- [Collect all tenants major compactions5,056476d45ea0fc5c,099706fd485e28ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.522 INFO 2458 --- [Collect all tenants major compactions6,056476d45ea0fc5c,fb2d44fe1ed2c78a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.522 INFO 2458 --- [Collect all tenants major compactions5,056476d45ea0fc5c,099706fd485e28ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:15:15.524 INFO 2458 --- [Collect all tenants major compactions6,056476d45ea0fc5c,fb2d44fe1ed2c78a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:15:15.531 INFO 2458 --- [Collect all tenants major compactions5,056476d45ea0fc5c,099706fd485e28ad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:15:15.533 INFO 2458 --- [Collect all tenants major compactions6,056476d45ea0fc5c,fb2d44fe1ed2c78a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:15:15.539 INFO 2458 --- [Collect all tenants major compactions4,056476d45ea0fc5c,2948a3e449a06aac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.540 INFO 2458 --- [Collect all tenants major compactions4,056476d45ea0fc5c,2948a3e449a06aac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:15:15.543 INFO 2458 --- [Collect all tenants major compactions4,056476d45ea0fc5c,2948a3e449a06aac] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:15:15.546 INFO 2458 --- [Collect all tenants major compactions2,056476d45ea0fc5c,75a94ac020cc4297] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.546 INFO 2458 --- [Collect all tenants major compactions3,056476d45ea0fc5c,f21efca1ca4b42e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.547 INFO 2458 --- [Collect all tenants major compactions2,056476d45ea0fc5c,75a94ac020cc4297] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:15:15.547 INFO 2458 --- [Collect all tenants major compactions3,056476d45ea0fc5c,f21efca1ca4b42e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:15:15.551 INFO 2458 --- [Collect all tenants major compactions3,056476d45ea0fc5c,f21efca1ca4b42e1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:15:15.551 INFO 2458 --- [Collect all tenants major compactions1,056476d45ea0fc5c,d95221b77d8dccf3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:15.551 INFO 2458 --- [Collect all tenants major compactions2,056476d45ea0fc5c,75a94ac020cc4297] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:15:15.552 INFO 2458 --- [Collect all tenants major compactions1,056476d45ea0fc5c,d95221b77d8dccf3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:15:15.555 INFO 2458 --- [Collect all tenants major compactions1,056476d45ea0fc5c,d95221b77d8dccf3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:15:15.556 INFO 2458 --- [ocp-schedules-7,056476d45ea0fc5c,f52db81806175865] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:15:16.213 INFO 2458 --- [ocp-schedules-9,c1c547259fd78842,c9d62db34d71b9a1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:16.536 INFO 2458 --- [ocp-async-5,cf16b350f1f666dd,511ff302e39daa68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:16.536 WARN 2458 --- [ocp-async-7,cd6fa682d74d1b7d,70848ca68b58ca53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:17.596 INFO 2458 --- [ocp-schedules-28,edd8e0fdd11ad35d,7fe0be42aa0ad6ab] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:15:17.596 INFO 2458 --- [ocp-schedules-28,edd8e0fdd11ad35d,7fe0be42aa0ad6ab] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:15:17.604 INFO 2458 --- [ocp-schedules-28,edd8e0fdd11ad35d,7fe0be42aa0ad6ab] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:15:17.604 INFO 2458 --- [ocp-schedules-28,edd8e0fdd11ad35d,7fe0be42aa0ad6ab] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:15:17.617 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534840, nodeCount=1000, affectRows=1000 2025-10-27 11:15:17.621 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534840, nodeCount=996, validCount=984 2025-10-27 11:15:17.707 INFO 2458 --- [prometheus-minute-schedule-1,271ebbcd48296efb,15272a8aa32f351f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:17.714 INFO 2458 --- [prometheus-minute-schedule-1,271ebbcd48296efb,15272a8aa32f351f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:18.155 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:18.156 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:18.516 INFO 2458 --- [ocp-async-3,bdaa8bac78fee592,97cb0e476437bc33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:18.536 INFO 2458 --- [ocp-async-2,2345d9c22afe6886,c0d0696432fc4248] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:18.536 WARN 2458 --- [ocp-async-4,1b60c4f99a150ac0,3eb2dad52461b3dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:19.638 INFO 2458 --- [ocp-schedules-20,5df688c9c13e7951,2aa98370dc98c5fb] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:15:20.537 WARN 2458 --- [ocp-async-1,58601d1a77722ecc,3567f43337c6271c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:20.537 INFO 2458 --- [ocp-async-9,f573e2343d00bede,d6cb4c6ad648b1ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:20.707 INFO 2458 --- [prometheus-minute-schedule-1,786210536b453cb8,11dd41839ca6ecaf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:20.715 INFO 2458 --- [prometheus-minute-schedule-1,786210536b453cb8,11dd41839ca6ecaf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:21.156 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:21.156 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:21.408 INFO 2458 --- [ocp-schedules-3,8a2874b54b4f4047,9a14fc2fde59bb4e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:21.408 INFO 2458 --- [ocp-schedules-23,edc33c2fcb20348f,5b219a837be11650] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:15:21.409 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:15:21.410 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:15:21.410 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:15:21.410 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:15:21.423 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:21.424 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:21.424 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:21.424 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:21.425 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:21.426 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:21.427 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:21.427 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:21.428 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:21.428 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:21.428 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:21.429 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:21.429 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:21.429 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:21.430 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:21.431 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:21.432 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:21.432 INFO 2458 --- [ocp-schedules-14,6255f0dfc3ff1441,a3c5c396722b3ec5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:21.432 INFO 2458 --- [host-check-clockdiff2,6255f0dfc3ff1441,0e1d368f2447402e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:15:21.432 INFO 2458 --- [host-check-clockdiff1,6255f0dfc3ff1441,c8d1b05465801b12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:15:21.433 INFO 2458 --- [host-check-clockdiff3,6255f0dfc3ff1441,7f6b67d0cc0c18a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:15:21.433 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:21.433 INFO 2458 --- [host-check-clockdiff4,6255f0dfc3ff1441,b34c701a61f0a340] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:15:21.433 INFO 2458 --- [host-check-clockdiff5,6255f0dfc3ff1441,a9f5457c9c92f2ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:15:21.434 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:21.436 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:21.454 INFO 2458 --- [ocp-schedules-7,7129c4e4935f6604,5a3ae0ab1729f6db] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:15:21.463 INFO 2458 --- [host-check-clockdiff2,6255f0dfc3ff1441,0e1d368f2447402e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534921 -1 -1 , error= 2025-10-27 11:15:21.465 INFO 2458 --- [host-check-clockdiff1,6255f0dfc3ff1441,c8d1b05465801b12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534921 0 0 , error= 2025-10-27 11:15:21.467 INFO 2458 --- [host-check-clockdiff4,6255f0dfc3ff1441,b34c701a61f0a340] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534921 -1 -1 , error= 2025-10-27 11:15:21.504 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:15:21.505 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:15:21.507 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:15:21.505752027+08:00 2025-10-27 11:15:21.507 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:15:21.508 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:15:21.507+08:00 2025-10-27 11:15:21.508 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:15:21.509 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:15:21.509 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:15:21.509 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:15:21.509 INFO 2458 --- [host-check-agent-status5,8a2874b54b4f4047,d84ba0a8d7bb4c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:15:21.509 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:15:21.508981081+08:00 2025-10-27 11:15:21.510 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:15:21.510 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:15:21.508918832+08:00 2025-10-27 11:15:21.511 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:15:21.509+08:00 2025-10-27 11:15:21.511 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:15:21.510+08:00 2025-10-27 11:15:21.512 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:15:21.512 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:15:21.512 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:15:21.512 INFO 2458 --- [host-check-agent-status3,8a2874b54b4f4047,d6d15f92cc557162] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:15:21.512 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:15:21.513 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:15:21.513 INFO 2458 --- [host-check-agent-status4,8a2874b54b4f4047,ce7585771cc781b8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:15:21.514 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:15:21.513933224+08:00 2025-10-27 11:15:21.514 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:15:21.514+08:00 2025-10-27 11:15:21.515 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:15:21.518 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:15:21.518 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:15:21.518 INFO 2458 --- [host-check-agent-status1,8a2874b54b4f4047,a36c9ca0253848ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:15:21.519 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:15:21.520 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:15:21.520 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:15:21.520138588+08:00 2025-10-27 11:15:21.520 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:15:21.520+08:00 2025-10-27 11:15:21.522 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:15:21.510+08:00 and status AVAILABLE 2025-10-27 11:15:21.522 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:15:21.507+08:00 and status AVAILABLE 2025-10-27 11:15:21.522 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:15:21.522 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:15:21.522 INFO 2458 --- [host-check-agent-status2,8a2874b54b4f4047,412ba2d3e5406539] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:15:21.525 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:15:21.514+08:00 and status AVAILABLE 2025-10-27 11:15:21.527 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:15:21.529 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:15:21.531 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:15:21.533 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:15:21.509+08:00 and status AVAILABLE 2025-10-27 11:15:21.535 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:15:21.520+08:00 and status AVAILABLE 2025-10-27 11:15:21.537 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:15:21.541 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:15:21.545 INFO 2458 --- [ocp-schedules-8,850a479d988ddd09,1f64bb3e95eb1e1e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:15:21.548 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:21.549 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:21.549 INFO 2458 --- [ocp-schedules-28,73d78016e60ee252,2c0d5a89200c7956] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:15:21.549 INFO 2458 --- [ocp-schedules-8,850a479d988ddd09,1f64bb3e95eb1e1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:21.550 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:21.551 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:21.551 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:21.552 INFO 2458 --- [ocp-schedules-28,73d78016e60ee252,2c0d5a89200c7956] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:21.553 INFO 2458 --- [ocp-schedules-28,73d78016e60ee252,2c0d5a89200c7956] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:15:21.553 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:21.554 INFO 2458 --- [ocp-schedules-4,722a529848799289,4c1367be8f0ab3ff] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:15:21.557 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:21.558 INFO 2458 --- [ocp-schedules-4,722a529848799289,4c1367be8f0ab3ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:21.560 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:21.563 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:21.566 INFO 2458 --- [ocp-schedules-1,880a55427d570eb3,db4ddd18fd159fb8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:15:21.566 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:21.587 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.588 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:21.589 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:15:21.608 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.609 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:21.610 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:21.612 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:21.615 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:15:21.620 INFO 2458 --- [ocp-schedules-8,850a479d988ddd09,1f64bb3e95eb1e1e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:15:21.621 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:21.625 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.625 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:21.625 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:21.630 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:15:21.632 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:15:21.637 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:15:21.645 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:15:21.649 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:15:21.668 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:15:21.669 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:15:21.669 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:15:21.682 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:15:21.682 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.683 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:21.685 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.685 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:15:21.689 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:15:21.707 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:21.708 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:21.709 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:21.715 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:21.714018+08:00, lastProcessTime=2025-10-27T11:15:11.726101+08:00 2025-10-27 11:15:21.724 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:15:21.729 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:15:21.730 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:15:21.734 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:21.736 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:15:21.760 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:21.761 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.761 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.765 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:15:21.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:15:21.767 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:15:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:21.769 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:15:21.773 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:15:21.774 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:21.775 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.775 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:15:21.781 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:15:21.792 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:21.792 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:21.797 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:21.798 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:21.799 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:15:21.803 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:21.813 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:15:21.816 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:21.817 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:15:21.818 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:21.823 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:15:21.835 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:15:21.841 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:21.849 INFO 2458 --- [ocp-schedules-26,6e9555fc0e8d9b5a,ba33ab804f2d4927] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:21.875 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:21.876 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:15:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:15:21.880 INFO 2458 --- [host-check-host-status3,edc33c2fcb20348f,9954b75d049bfc34] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:15:21.881 INFO 2458 --- [host-check-host-status4,edc33c2fcb20348f,5018fe44fc99681e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:15:21.884 INFO 2458 --- [host-check-host-status2,edc33c2fcb20348f,821550a06889dbfd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:15:21.891 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:15:21.892 INFO 2458 --- [host-check-host-status5,edc33c2fcb20348f,97bb5c29ae755226] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:15:21.904 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:21.904 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.905 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:21.906 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:21.911 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:15:21.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:21.914 WARN 2458 --- [scanner-112157524-1,6e9555fc0e8d9b5a,4c3646757b5e1f64] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:21.931 INFO 2458 --- [host-check-host-status1,edc33c2fcb20348f,abb92addd46fe926] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:15:21.935 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:21.936 INFO 2458 --- [alarm-detect-17,6e9555fc0e8d9b5a,170b36a0d996a669] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:21.938 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:21.940 INFO 2458 --- [alarm-detect-17,6e9555fc0e8d9b5a,170b36a0d996a669] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:21.954 INFO 2458 --- [alarm-detect-17,6e9555fc0e8d9b5a,a7464f0e01acf3ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:21.954 INFO 2458 --- [alarm-detect-2,6e9555fc0e8d9b5a,86417c7c1b7520b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:21.958 INFO 2458 --- [alarm-detect-17,6e9555fc0e8d9b5a,a7464f0e01acf3ed] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:21.959 INFO 2458 --- [alarm-detect-2,6e9555fc0e8d9b5a,86417c7c1b7520b5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:21.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:21.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:22.003 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:15:22.030 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:21.714018+08:00 2025-10-27 11:15:22.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:22.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:22.047 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:15:22.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:15:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.051 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:15:22.063 INFO 2458 --- [alarm-detect-13,6e9555fc0e8d9b5a,12b5d3c21128d141] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:22.067 INFO 2458 --- [alarm-detect-13,6e9555fc0e8d9b5a,12b5d3c21128d141] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:22.076 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:15:22.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:22.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:22.142 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:22.142 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.143 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:15:22.166 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:22.166 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:22.167 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:15:22.167 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:15:22.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:22.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:22.233 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:22.261 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f562a6c93df8fcae,2c52d0b1fcd01a57] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=f562a6c93df8fcae, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:15:22.261 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,f562a6c93df8fcae,2c52d0b1fcd01a57] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=f562a6c93df8fcae, duration=0 ms] 2025-10-27 11:15:22.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:22.274 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.274 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:15:22.287 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:15:22.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:22.299 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:22.346 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:15:22.382 INFO 2458 --- [host-check-clockdiff5,6255f0dfc3ff1441,a9f5457c9c92f2ee] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534922 0 0 , error= 2025-10-27 11:15:22.419 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:15:22.420 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.421 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:15:22.423 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.423 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:15:22.460 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:15:22.501 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:22.502 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:22.502 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:15:22.537 INFO 2458 --- [ocp-async-10,92af846f3efc37b7,c1f9f8abf62aca94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:22.537 WARN 2458 --- [ocp-async-7,28694095c7979ffc,60c9eb5307cef8fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:22.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:15:22.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:15:22.553 INFO 2458 --- [ocp-schedules-4,722a529848799289,4c1367be8f0ab3ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:15:23.232 INFO 2458 --- [host-check-clockdiff3,6255f0dfc3ff1441,7f6b67d0cc0c18a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534923 0 0 , error= 2025-10-27 11:15:23.707 INFO 2458 --- [prometheus-minute-schedule-1,0b6ac780008eb2d1,b50b5001e1c417c7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:23.715 INFO 2458 --- [prometheus-minute-schedule-1,0b6ac780008eb2d1,b50b5001e1c417c7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:24.157 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:24.158 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:24.161 INFO 2458 --- [ocp-schedules-3,4f7966dbb2820fc7,1c94814a6d9aaa6d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:24.165 INFO 2458 --- [ocp-schedules-3,4f7966dbb2820fc7,1c94814a6d9aaa6d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:24.537 INFO 2458 --- [ocp-async-4,b6197680da8e4804,098341c63e6b7304] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:24.537 WARN 2458 --- [ocp-async-8,dd6fe0b1a1ae1d2b,0cb76b832cf195c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:26.221 INFO 2458 --- [ocp-schedules-11,fd2fc32cbf3df482,d8b8e448ca1c85c6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:26.537 INFO 2458 --- [ocp-async-2,d85fbee0e23fe4a2,cc8942ab88099bdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:26.537 WARN 2458 --- [ocp-async-6,5d7f0bff4d473f13,3b5d4b847390774b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:26.707 INFO 2458 --- [prometheus-minute-schedule-1,95b5cdfaf6456af6,8db6fa997261535c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:26.714 INFO 2458 --- [prometheus-minute-schedule-1,95b5cdfaf6456af6,8db6fa997261535c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:26.991 INFO 2458 --- [ocp-schedules-5,39ea111b300febf5,4b7885bfe9e8a87e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:15:26.999 INFO 2458 --- [ocp-schedules-5,39ea111b300febf5,4b7885bfe9e8a87e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:26.999 INFO 2458 --- [ocp-schedules-5,39ea111b300febf5,4b7885bfe9e8a87e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:27.158 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:27.159 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:28.517 INFO 2458 --- [ocp-async-2,958e972e3602b8c0,1733d083291168e8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:28.537 INFO 2458 --- [ocp-async-9,6204a09ba082b1f5,72769ee6e27d3682] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:28.537 WARN 2458 --- [ocp-async-7,5dfa625e13d88dc5,8a7c132451a448f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:28.904 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:15:28.904 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:15:28.909 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:15:28.909 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260515940638720 2025-10-27 11:15:28.914 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=839260767611461632 2025-10-27 11:15:28.914 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:15:28.914 INFO 2458 --- [ocp-schedules-20,883545f65b8f669f,a22dc3f739e8c68c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:15:29.372 INFO 2458 --- [ocp-schedules-3,53e197fbd68cf560,24472fc7f4e9dd6f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:15:29.376 INFO 2458 --- [ocp-schedules-3,53e197fbd68cf560,24472fc7f4e9dd6f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:15:29.376 INFO 2458 --- [ocp-schedules-3,53e197fbd68cf560,24472fc7f4e9dd6f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:15:29.707 INFO 2458 --- [prometheus-minute-schedule-1,290f738f5b8a4cd6,768fe6379a394985] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:29.714 INFO 2458 --- [prometheus-minute-schedule-1,290f738f5b8a4cd6,768fe6379a394985] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:30.159 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:30.161 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:30.537 INFO 2458 --- [ocp-async-9,52617f9830d85a09,105a8692903720f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:30.537 WARN 2458 --- [ocp-async-8,dd4a7055e60f2378,2a6910d64b12987b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:31.714352+08:00, lastProcessTime=2025-10-27T11:15:21.714018+08:00 2025-10-27 11:15:31.936 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:31.714352+08:00 2025-10-27 11:15:32.135 INFO 2458 --- [ocp-schedules-2,ad31744a59898c8f,87b847f106d7b23d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:32.159 WARN 2458 --- [scanner-112157767-1,ad31744a59898c8f,4fcf32fdb2a04045] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:32.258 INFO 2458 --- [alarm-detect-16,ad31744a59898c8f,2e87c28103b5344f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534900 ~ 1761534922 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534915, value=55.671)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534915, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534915=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:15:32.537 WARN 2458 --- [ocp-async-9,a53b6c154109f879,a21ea169083ef4f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:32.537 INFO 2458 --- [ocp-async-9,5245e5f87e75350b,51d9231c1beacd19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:32.707 INFO 2458 --- [prometheus-minute-schedule-1,3c13c06c8ad1c386,9978f3854e7fe0ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:32.715 INFO 2458 --- [prometheus-minute-schedule-1,3c13c06c8ad1c386,9978f3854e7fe0ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:33.161 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:33.162 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:33.806 INFO 2458 --- [ocp-schedules-13,186aadfe466dd03c,3dadb0cc64120925] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:15:33.806 INFO 2458 --- [ocp-schedules-13,186aadfe466dd03c,3dadb0cc64120925] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:15:34.112 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:15:34.112 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:15:34.131 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:15:34.131 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:15:34.131 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:15:34.145 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:15:34.145 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:15:34.145 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:15:34.158 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:15:34.158 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:15:34.158 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:15:34.167 INFO 2458 --- [ocp-schedules-5,b674de0c55a4efb2,fc20f4895f3dc997] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:15:34.172 INFO 2458 --- [ocp-schedules-28,1675a659e916173a,9ba3dd08d4ca06e3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:34.176 INFO 2458 --- [ocp-schedules-28,1675a659e916173a,9ba3dd08d4ca06e3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:34.538 WARN 2458 --- [ocp-async-10,b202d80d4abd00e4,b14c3878d13ef57d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:34.538 INFO 2458 --- [ocp-async-2,811750529f3d594e,3118315e4276491d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:35.707 INFO 2458 --- [prometheus-minute-schedule-1,5f060de596fce674,980e820495d3c2a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:35.715 INFO 2458 --- [prometheus-minute-schedule-1,5f060de596fce674,980e820495d3c2a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:36.162 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:36.163 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:36.228 INFO 2458 --- [ocp-schedules-10,ddc4b64cdbe668f7,1e727c2cb0a90ed5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:36.538 WARN 2458 --- [ocp-async-4,8157c5a6ffa12150,096776174136f5ea] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:36.538 INFO 2458 --- [ocp-async-8,6216e069f84cdf99,04c189432a5b59ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:36.619 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:15:36.619 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:15:36.720 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:36.721 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:15:36.733 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:36.734 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:15:36.796 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:36.797 INFO 2458 --- [ocp-schedules-11,bb0d3c3104f56e33,c3ca1a675d0d80d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:15:37.708 INFO 2458 --- [prometheus-cache-monitor1,469d5d00f00f538c,f8dc90c42ee1be8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20335,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5098,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:15:37.708 INFO 2458 --- [prometheus-cache-monitor1,469d5d00f00f538c,f8dc90c42ee1be8d] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:15:38.009 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:38.009 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:15:38.009 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:15:38.009 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:15:38.009 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:15:38.010 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:15:38.019 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:38.022 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:38.022 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:38.022 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:38.022 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:38.024 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:38.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:38.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:38.026 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:38.026 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:38.079 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:15:38.081 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:15:38.083 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:15:38.084 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:15:38.085 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:15:38.090 INFO 2458 --- [ocp-schedules-15,eeb57e349b4e25d4,b342b71e6c07b622] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:15:38.517 INFO 2458 --- [ocp-async-8,9df8779a1f24c290,e3cfdd87110c7932] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:38.538 INFO 2458 --- [ocp-async-1,e26378e435d87cbd,7c2c469cf2024e03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:38.538 WARN 2458 --- [ocp-async-7,9a493f6543368a63,fd4e88ad34e687b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:38.651 INFO 2458 --- [ocp-schedules-23,27dc659bc0313c91,f5303e2a72a3b8a1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:15:38.661 INFO 2458 --- [ocp-schedules-23,27dc659bc0313c91,f5303e2a72a3b8a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:38.661 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:15:38.662 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:15:38.663 INFO 2458 --- [ocp-schedules-23,27dc659bc0313c91,f5303e2a72a3b8a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:15:38.707 INFO 2458 --- [prometheus-minute-schedule-1,da2c0f23ede916f8,37d0756e1f23c81d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:38.714 INFO 2458 --- [prometheus-minute-schedule-1,da2c0f23ede916f8,37d0756e1f23c81d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:39.164 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:39.165 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:39.221 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,d3017a10ed43ae4d,f604d83c11b15f14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=d3017a10ed43ae4d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:15:39.221 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,d3017a10ed43ae4d,f604d83c11b15f14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=d3017a10ed43ae4d, duration=1 ms] 2025-10-27 11:15:40.012 INFO 2458 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:15:40.013 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:15:40.049 INFO 2458 --- [Collect obproxy1,e20348454f16cc46,dd50b1bfd8f881bd] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:15:40.049 INFO 2458 --- [Collect obproxy1,e20348454f16cc46,dd50b1bfd8f881bd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:15:40.049 INFO 2458 --- [Check obproxy1,341f08c3f50094c3,8693f7c9149e096c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:15:40.049 INFO 2458 --- [Check obproxy1,341f08c3f50094c3,8693f7c9149e096c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:15:40.077 INFO 2458 --- [Check obproxy1,341f08c3f50094c3,8693f7c9149e096c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:15:40.078 INFO 2458 --- [Collect obproxy1,e20348454f16cc46,dd50b1bfd8f881bd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:15:40.079 INFO 2458 --- [Check obproxy1,341f08c3f50094c3,8693f7c9149e096c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:15:40.079 INFO 2458 --- [Check obproxy1,341f08c3f50094c3,8693f7c9149e096c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:15:40.081 INFO 2458 --- [Collect obproxy1,e20348454f16cc46,dd50b1bfd8f881bd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:15:40.084 INFO 2458 --- [Collect obproxy1,e20348454f16cc46,dd50b1bfd8f881bd] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:15:40.538 WARN 2458 --- [ocp-async-1,7f83898106c363d1,eec4bc477ce3e346] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:40.538 INFO 2458 --- [ocp-async-9,08ad90bf259aaf2d,519d70229704b3b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:40.929 INFO 2458 --- [ocp-schedules-19,c09de094dc35e80e,dddf5b64de0326a2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:15:40.930 INFO 2458 --- [ocp-schedules-19,c09de094dc35e80e,dddf5b64de0326a2] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:15:41.707 INFO 2458 --- [prometheus-minute-schedule-1,2a76dbae68263322,7e2d4db1962c4eff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:41.714 INFO 2458 --- [prometheus-minute-schedule-1,2a76dbae68263322,7e2d4db1962c4eff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:41.713610+08:00, lastProcessTime=2025-10-27T11:15:31.714352+08:00 2025-10-27 11:15:41.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:41.713610+08:00 2025-10-27 11:15:42.165 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:42.166 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:42.271 INFO 2458 --- [ocp-schedules-25,2e7a7788f23be9fd,55e468ca44ef2256] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:42.317 WARN 2458 --- [scanner-112157963-1,2e7a7788f23be9fd,c30750db201e6676] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:42.329 INFO 2458 --- [alarm-detect-19,2e7a7788f23be9fd,841b9e45cf477395] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:42.333 INFO 2458 --- [alarm-detect-19,2e7a7788f23be9fd,841b9e45cf477395] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:15:42.345 INFO 2458 --- [alarm-detect-17,2e7a7788f23be9fd,7491f9326fe9473b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:42.350 INFO 2458 --- [alarm-detect-17,2e7a7788f23be9fd,7491f9326fe9473b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:15:42.353 INFO 2458 --- [alarm-detect-16,2e7a7788f23be9fd,5a1f28025062dd90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:42.357 INFO 2458 --- [alarm-detect-16,2e7a7788f23be9fd,5a1f28025062dd90] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:15:42.422 INFO 2458 --- [alarm-detect-15,2e7a7788f23be9fd,82b07f2a897c5f0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:42.426 INFO 2458 --- [alarm-detect-15,2e7a7788f23be9fd,82b07f2a897c5f0f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:15:42.486 INFO 2458 --- [alarm-detect-20,2e7a7788f23be9fd,95f53a3fa0d913f3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534915 ~ 1761534932 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534930, value=75.37)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534930, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534930=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:15:42.538 WARN 2458 --- [ocp-async-9,8a2eca6b01fecde4,b8baeebead2040d2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:42.538 INFO 2458 --- [ocp-async-3,7cab3339e6163d8d,248cb0f7c6cec8e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:44.065 INFO 2458 --- [ocp-schedules-6,b126ed27275729cd,7ed9814f895a9a13] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:15:44.192 INFO 2458 --- [ocp-schedules-2,8a1ceac818f371b0,8d177d931e67cd56] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:44.195 INFO 2458 --- [ocp-schedules-2,8a1ceac818f371b0,8d177d931e67cd56] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:44.538 WARN 2458 --- [ocp-async-8,d87044452d11f4e6,eabaa922423ed52c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:44.538 INFO 2458 --- [ocp-async-6,a10816761bdf7622,5c5779b6ef1c947e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:44.707 INFO 2458 --- [prometheus-minute-schedule-1,05c4ca18fbce0b59,8ad439beb4e9f36b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:44.715 INFO 2458 --- [prometheus-minute-schedule-1,05c4ca18fbce0b59,8ad439beb4e9f36b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:45.102 INFO 2458 --- [prometheus-second-schedule-1,5d26b8d44649d27a,34be90956cc39ca1] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:15:45.166 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:45.167 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:46.234 INFO 2458 --- [ocp-schedules-6,77c94e214f3a27a3,153073d5262e17ea] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:46.538 WARN 2458 --- [ocp-async-6,aebc37f25ae84a36,c0ada347cba6904d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:46.538 INFO 2458 --- [ocp-async-5,9e25ce8f489c4e3e,9a504af5a4558dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:47.002 INFO 2458 --- [ocp-schedules-30,63bed77eae921621,3b43a48211eea018] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:15:47.010 INFO 2458 --- [ocp-schedules-30,63bed77eae921621,3b43a48211eea018] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:47.010 INFO 2458 --- [ocp-schedules-30,63bed77eae921621,3b43a48211eea018] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:15:47.707 INFO 2458 --- [prometheus-minute-schedule-1,a3cb2635b257dc2b,802e3ea2fa030bb8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:47.715 INFO 2458 --- [prometheus-minute-schedule-1,a3cb2635b257dc2b,802e3ea2fa030bb8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:47.749 INFO 2458 --- [ocp-schedules-5,86d38f0446823e10,d6c991c16f128502] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:15:47.749 INFO 2458 --- [ocp-schedules-5,86d38f0446823e10,d6c991c16f128502] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:15:47.756 INFO 2458 --- [ocp-schedules-5,86d38f0446823e10,d6c991c16f128502] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:15:47.756 INFO 2458 --- [ocp-schedules-5,86d38f0446823e10,d6c991c16f128502] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:15:48.167 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:48.167 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:48.480 INFO 2458 --- [ocp-schedules-19,e03ca5748e7ac587,ad36e3a710fd55c4] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:15:48.480+08:00, clusters=[1, 2] 2025-10-27 11:15:48.495 INFO 2458 --- [ocp-schedules-19,e03ca5748e7ac587,ad36e3a710fd55c4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:15:48.498 INFO 2458 --- [ocp-schedules-19,e03ca5748e7ac587,ad36e3a710fd55c4] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:15:48.517 INFO 2458 --- [ocp-async-4,ab85532407a93549,27b99377437769a0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:48.539 WARN 2458 --- [ocp-async-2,3b16cb14b17cc88a,87f33c9c5c6c9c1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:48.539 INFO 2458 --- [ocp-async-3,d8dde195a93d0798,4664c5b60202d639] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:50.539 INFO 2458 --- [ocp-async-8,195a0833ae7f0ac1,112e36776fe53a89] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:50.539 WARN 2458 --- [ocp-async-10,dbd82eaab21ce9fc,1e9ec0e96502e07c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:50.707 INFO 2458 --- [prometheus-minute-schedule-1,65d915aaf47cc7b0,bb516380efeda494] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:50.715 INFO 2458 --- [prometheus-minute-schedule-1,65d915aaf47cc7b0,bb516380efeda494] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:51.168 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:51.168 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:51.411 INFO 2458 --- [ocp-schedules-10,7cb4c2035872c70f,b36f56b1e7640afc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:51.411 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:15:51.411 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:15:51.412 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:15:51.411 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:15:51.412 INFO 2458 --- [ocp-schedules-8,9edae0897f6657b4,09c720971d3619af] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:51.412 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:15:51.413 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:15:51.413 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:15:51.413 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:15:51.413 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:15:51.413 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:15:51.428 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:51.429 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:51.429 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:51.430 INFO 2458 --- [ocp-schedules-24,6b519df7df377469,c15181faadd1927e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:15:51.430 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:51.430 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:51.430 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:51.431 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:51.432 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:51.432 INFO 2458 --- [host-check-clockdiff1,6b519df7df377469,acf40e6656ce04de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:15:51.432 INFO 2458 --- [host-check-clockdiff2,6b519df7df377469,114d3c157aadd472] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:15:51.433 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:51.433 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:51.433 INFO 2458 --- [host-check-clockdiff3,6b519df7df377469,b6a076c1cdd97693] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:15:51.434 INFO 2458 --- [host-check-clockdiff4,6b519df7df377469,e57962172f5ed9bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:15:51.434 INFO 2458 --- [host-check-clockdiff5,6b519df7df377469,eb1105b89315569e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:15:51.434 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:51.435 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:51.435 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:51.435 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:51.435 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:51.436 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:51.436 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:51.442 INFO 2458 --- [ocp-schedules-6,df44c68a692717ed,a57b30ec105fc92d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:15:51.459 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:51.459 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:51.456 INFO 2458 --- [host-check-clockdiff4,6b519df7df377469,e57962172f5ed9bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534951 -1 -1 , error= 2025-10-27 11:15:51.466 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:51.467 INFO 2458 --- [host-check-clockdiff1,6b519df7df377469,acf40e6656ce04de] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534951 0 0 , error= 2025-10-27 11:15:51.467 INFO 2458 --- [host-check-clockdiff3,6b519df7df377469,b6a076c1cdd97693] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534951 0 0 , error= 2025-10-27 11:15:51.519 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:15:51.519 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:15:51.520 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:15:51.521 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:15:51.521 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:15:51.519915138+08:00 2025-10-27 11:15:51.521 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:15:51.521 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:15:51.522 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:15:51.522028893+08:00 2025-10-27 11:15:51.522 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:15:51.521+08:00 2025-10-27 11:15:51.522 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:15:51.522+08:00 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status3,9edae0897f6657b4,b475d15291ecc852] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:15:51.523 INFO 2458 --- [host-check-agent-status2,9edae0897f6657b4,834bf96b5f7ce36a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status4,9edae0897f6657b4,40b12da864d580bc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:15:51.525 INFO 2458 --- [host-check-agent-status1,9edae0897f6657b4,6bade965102fe9d6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:15:51.529 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:15:51.530 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:15:51.522+08:00 and status AVAILABLE 2025-10-27 11:15:51.530 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:15:51.530 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:15:51.521+08:00 and status AVAILABLE 2025-10-27 11:15:51.531 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:15:51.531101747+08:00 2025-10-27 11:15:51.531 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:15:51.531 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:15:51.530867974+08:00 2025-10-27 11:15:51.531 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:15:51.531+08:00 2025-10-27 11:15:51.532 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:15:51.531+08:00 2025-10-27 11:15:51.533 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:15:51.533 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:15:51.533 INFO 2458 --- [host-check-agent-status5,9edae0897f6657b4,6605149153b0ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:15:51.534 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:15:51.534 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:15:51.534 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:15:51.534047603+08:00 2025-10-27 11:15:51.535 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:15:51.534+08:00 2025-10-27 11:15:51.536 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:15:51.538 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:15:51.531+08:00 and status AVAILABLE 2025-10-27 11:15:51.539 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:15:51.531+08:00 and status AVAILABLE 2025-10-27 11:15:51.542 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:15:51.534+08:00 and status AVAILABLE 2025-10-27 11:15:51.543 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:15:51.546 INFO 2458 --- [ocp-schedules-26,837c0e843f9727c2,033b97528b2c247b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:15:51.546 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:15:51.549 INFO 2458 --- [ocp-schedules-7,69f2db2e9be48aae,c1c7615f5002082a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:15:51.549 INFO 2458 --- [ocp-schedules-26,837c0e843f9727c2,033b97528b2c247b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:51.549 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:15:51.553 INFO 2458 --- [ocp-schedules-7,69f2db2e9be48aae,c1c7615f5002082a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:51.554 INFO 2458 --- [ocp-schedules-17,32adef763fdde0d0,82dd590e6cce5e06] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:15:51.555 INFO 2458 --- [ocp-schedules-7,69f2db2e9be48aae,c1c7615f5002082a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:15:51.555 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:51.557 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:15:51.559 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:51.560 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:15:51.560 INFO 2458 --- [ocp-schedules-17,32adef763fdde0d0,82dd590e6cce5e06] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:51.563 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:51.566 INFO 2458 --- [ocp-schedules-12,c73edba0c15a5972,23083ee985fcd75b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:15:51.566 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:51.568 INFO 2458 --- [ocp-schedules-16,e7f69ea64f73139a,19f1c7cb60e22114] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:15:51.569 INFO 2458 --- [ocp-schedules-15,beed9dc414cdec79,5ae9952e46fd1953] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:15:51.576 INFO 2458 --- [ocp-schedules-16,e7f69ea64f73139a,19f1c7cb60e22114] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:15:51.579 INFO 2458 --- [ocp-schedules-15,beed9dc414cdec79,5ae9952e46fd1953] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:15:51.586 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:15:51.588 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:15:51.590 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:15:51.591 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:15:51.621 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.622 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:51.623 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:15:51.644 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:51.649 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.649 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:51.658 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:15:51.659 INFO 2458 --- [ocp-schedules-26,837c0e843f9727c2,033b97528b2c247b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:15:51.661 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:51.675 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:15:51.676 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.677 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:51.679 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:51.689 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:51.691 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:15:51.699 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:15:51.706 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:15:51.707 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:15:51.709 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:15:51.709 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:15:51.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:15:51.715451+08:00, lastProcessTime=2025-10-27T11:15:41.713610+08:00 2025-10-27 11:15:51.726 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.726 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:51.735 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:51.737 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.737 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:15:51.739 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:15:51.740 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:15:51.740 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:15:51.744 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:15:51.744 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:51.780 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:15:51.785 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:51.786 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:51.786 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:51.787 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:15:51.801 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:15:51.803 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:15:51.805 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:15:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.807 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:51.808 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.808 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:15:51.811 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:15:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:15:51.813 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.816 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:15:51.832 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:15:51.837 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:51.837 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:51.840 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:15:51.846 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:15:51.849 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:15:51.854 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:51.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:51.875 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:51.886 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:15:51.888 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:51.890 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:15:51.892 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:51.901 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:15:51.905 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:15:51.907 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:51.908 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.908 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:51.909 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:15:51.911 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:15:51.921 INFO 2458 --- [host-check-host-status1,7cb4c2035872c70f,5e5e104eb5c64598] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:15:51.922 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:15:51.922 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:15:51.925 INFO 2458 --- [host-check-host-status4,7cb4c2035872c70f,13b502c15fd30c2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:15:51.928 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:15:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:51.934 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:15:51.937 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:15:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:15:51.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:51.947 INFO 2458 --- [host-check-host-status5,7cb4c2035872c70f,7f825bdd3f904561] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:15:51.950 INFO 2458 --- [host-check-host-status3,7cb4c2035872c70f,427b5e1e63889b07] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:15:51.955 INFO 2458 --- [host-check-host-status2,7cb4c2035872c70f,c14412545fe8600f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:15:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:51.991 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:15:51.715451+08:00 2025-10-27 11:15:51.995 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:15:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:52.036 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:15:52.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.038 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:15:52.039 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.040 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:15:52.058 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:15:52.060 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:52.060 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:52.081 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:52.082 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.083 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:15:52.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:52.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:15:52.104 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:15:52.104 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:15:52.130 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:52.130 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:15:52.164 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:15:52.206 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:52.206 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.207 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:15:52.210 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:15:52.222 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:15:52.231 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:15:52.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:15:52.282 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:15:52.348 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:15:52.349 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.349 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:15:52.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.352 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:15:52.382 INFO 2458 --- [host-check-clockdiff2,6b519df7df377469,114d3c157aadd472] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534952 -1 -1 , error= 2025-10-27 11:15:52.390 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:15:52.439 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:15:52.440 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:15:52.440 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:15:52.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:15:52.478 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:15:52.478 INFO 2458 --- [ocp-schedules-17,32adef763fdde0d0,82dd590e6cce5e06] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:15:52.491 INFO 2458 --- [ocp-schedules-1,453fe25be3e23082,85a80615b25cff5c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:15:52.513 WARN 2458 --- [scanner-112158230-1,453fe25be3e23082,86ae594cd73524b0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:15:52.539 WARN 2458 --- [ocp-async-4,ec3267584ec65c7d,9fb994e92e0ba7dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:52.539 INFO 2458 --- [ocp-async-4,7d3001c98570e52e,b617a643c35b5573] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:15:53.222 INFO 2458 --- [ocp-schedules-30,41d7ccb10dbe78c1,9d5fbe752213fb4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20335,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5098,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:15:53.222 INFO 2458 --- [ocp-schedules-30,41d7ccb10dbe78c1,9d5fbe752213fb4f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:15:53.707 INFO 2458 --- [prometheus-minute-schedule-1,718a12c6d4a28f72,c3fe96c4d6bb8b08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:53.714 INFO 2458 --- [prometheus-minute-schedule-1,718a12c6d4a28f72,c3fe96c4d6bb8b08] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:53.960 INFO 2458 --- [host-check-clockdiff5,6b519df7df377469,eb1105b89315569e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534953 0 0 , error= 2025-10-27 11:15:54.168 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:54.170 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:54.203 INFO 2458 --- [ocp-schedules-13,8b452800d14086be,853472e5629753ab] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:15:54.205 INFO 2458 --- [ocp-schedules-13,8b452800d14086be,853472e5629753ab] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:15:54.539 WARN 2458 --- [ocp-async-9,6e42432e3105b8f3,7b7fcc128f3fe3dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:54.539 INFO 2458 --- [ocp-async-3,a716ec9bbf2f4162,bebb91cde5973860] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:54.794 INFO 2458 --- [ocp-schedules-30,dd19daea687f31af,233c4b51d7927ca5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:15:56.241 INFO 2458 --- [ocp-schedules-11,c988679e84719e69,e29882fb6fa7d935] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:15:56.539 INFO 2458 --- [ocp-async-7,2ebced8c7a794c3a,58620734747a48d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:56.539 WARN 2458 --- [ocp-async-1,c8ec221cc0111fd8,90f4fa3f1f1ead25] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:56.707 INFO 2458 --- [prometheus-minute-schedule-1,cc618bb69ec3a733,e9100ae800fdd5d0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:56.714 INFO 2458 --- [prometheus-minute-schedule-1,cc618bb69ec3a733,e9100ae800fdd5d0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:15:57.170 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:15:57.171 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:15:57.893 INFO 2458 --- [ocp-schedules-11,3e64d282ddc31bf2,896e018cb485381f] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:15:57.902 INFO 2458 --- [ocp-schedules-11,3e64d282ddc31bf2,896e018cb485381f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:15:57.946 INFO 2458 --- [ocp-schedules-11,3e64d282ddc31bf2,896e018cb485381f] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:15:58.517 INFO 2458 --- [ocp-async-6,547bcc92064a763e,73fcc76709338d34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:15:58.539 WARN 2458 --- [ocp-async-5,89afb729d7427b6d,076b72948941dd57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:15:58.539 INFO 2458 --- [ocp-async-2,92fe3abf3aafbe02,9c121769de58efaf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:15:59.289 INFO 2458 --- [ocp-schedules-25,3b326fef0205ebcf,c1ce93d41703d30a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:15:59.593 INFO 2458 --- [ocp-schedules-7,33152c41a7fec441,3f949b561c55f90e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:15:59.594 INFO 2458 --- [ocp-schedules-7,33152c41a7fec441,3f949b561c55f90e] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:15:59.707 INFO 2458 --- [prometheus-minute-schedule-1,b725b43aab5edc4a,b7df5c2379f0e770] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761534960, interval=60 2025-10-27 11:15:59.714 INFO 2458 --- [prometheus-minute-schedule-1,b725b43aab5edc4a,b7df5c2379f0e770] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:00.171 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:00.172 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:00.539 INFO 2458 --- [ocp-async-10,207a0a32d3c43a11,60a623bbebdc8218] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:00.539 WARN 2458 --- [ocp-async-9,a379e30afcd1adb9,e48a4e14068ba1df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:01.713991+08:00, lastProcessTime=2025-10-27T11:15:51.715451+08:00 2025-10-27 11:16:01.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:01.713991+08:00 2025-10-27 11:16:02.540 INFO 2458 --- [ocp-async-1,a9df09ca5d106cce,a49691b75cb813f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:02.540 WARN 2458 --- [ocp-async-8,0940e6c73c099850,c9d4240ee1667aff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:02.624 INFO 2458 --- [ocp-schedules-23,30f173f3395ffc4b,47900d147fd6a5cf] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:02.707 INFO 2458 --- [prometheus-minute-schedule-1,d4b5cd5784411b39,f56c0ddd2efe0a71] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:02.714 INFO 2458 --- [prometheus-minute-schedule-1,d4b5cd5784411b39,f56c0ddd2efe0a71] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:02.766 WARN 2458 --- [scanner-112158418-1,30f173f3395ffc4b,a7a6d8e161943a4d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:02.783 INFO 2458 --- [alarm-detect-3,30f173f3395ffc4b,e719e5d99dd950bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:02.787 INFO 2458 --- [alarm-detect-3,30f173f3395ffc4b,e719e5d99dd950bc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:02.953 INFO 2458 --- [ocp-schedules-15,14880c138593ec34,e2a7b1efc3f5266d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:16:02.961 INFO 2458 --- [ocp-schedules-15,14880c138593ec34,e2a7b1efc3f5266d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:02.964 INFO 2458 --- [alarm-detect-2,30f173f3395ffc4b,38d1eeb0ad40a88d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:02.964 INFO 2458 --- [alarm-detect-16,30f173f3395ffc4b,b476269cd7684e59] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:02.968 INFO 2458 --- [alarm-detect-2,30f173f3395ffc4b,38d1eeb0ad40a88d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:02.969 INFO 2458 --- [alarm-detect-16,30f173f3395ffc4b,b476269cd7684e59] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:03.014 INFO 2458 --- [ocp-schedules-11,a83647dc4e9d7f5c,f0b17739a5abb528] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:16:03.018 INFO 2458 --- [ocp-schedules-11,a83647dc4e9d7f5c,f0b17739a5abb528] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:03.025 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.026 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:16:03.057 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.058 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:16:03.169 INFO 2458 --- [ocp-schedules-15,14880c138593ec34,e2a7b1efc3f5266d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:16:03.175 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:03.176 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:03.211 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:16:03.211 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.216 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:16:03.225 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.225 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.247 INFO 2458 --- [alarm-detect-14,30f173f3395ffc4b,eab1501c88241563] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:03.252 INFO 2458 --- [alarm-detect-14,30f173f3395ffc4b,eab1501c88241563] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:03.257 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.258 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.280 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:16:03.281 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.286 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.287 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.288 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:16:03.289 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.290 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.322 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.322 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.349 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.349 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.371 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.372 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.413 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.414 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.415 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.416 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.417 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.418 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.419 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.424 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.425 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.426 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.426 INFO 2458 --- [alarm-detect-15,30f173f3395ffc4b,8679d2dcf78e5662] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534930 ~ 1761534953 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534945, value=66.617)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534945, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534945=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:16:03.429 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.430 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.430 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:16:03.434 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:16:03.448 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.449 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:16:03.453 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.454 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.489 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.490 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.494 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:03.495 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.495 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:16:03.505 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.505 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:16:03.505 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.505 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:16:03.508 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:16:03.508 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:16:03.522 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.523 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:16:03.526 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.527 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:16:03.561 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.561 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:16:03.595 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.596 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:16:03.599 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.600 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:16:03.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:16:03.659 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.660 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:16:03.710 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.711 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.712 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.713 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.713 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.714 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.715 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.716 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.716 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.717 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.717 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.718 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.719 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.719 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.720 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.722 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:16:03.724 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.725 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:16:03.725 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.726 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:16:03.726 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:16:03.729 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:16:03.815 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:03.815 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:03.816 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:16:03.842 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:16:03.846 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:16:03.846 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:16:03.846 INFO 2458 --- [ocp-schedules-11,a83647dc4e9d7f5c,f0b17739a5abb528] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:16:04.213 INFO 2458 --- [ocp-schedules-6,dfc74f5314dba2d0,71d7639ca877e29f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:04.217 INFO 2458 --- [ocp-schedules-6,dfc74f5314dba2d0,71d7639ca877e29f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:04.540 INFO 2458 --- [ocp-async-1,264e6c1e606ed3c1,3daa653826bc8b56] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:04.540 WARN 2458 --- [ocp-async-6,43ff5b61cda70576,1c1eb6318bb4bbb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:05.707 INFO 2458 --- [prometheus-minute-schedule-1,d30558a61bb53246,58a4128d7dacb9a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:05.715 INFO 2458 --- [prometheus-minute-schedule-1,d30558a61bb53246,58a4128d7dacb9a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:16:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:06.003 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:16:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:16:06.054 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:16:06.094 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:16:06.116 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:16:06.116 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:16:06.116 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:16:06.116 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:16:06.119 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:16:06.120 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:06.120 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:16:06.122 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:06.123 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:16:06.176 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:06.177 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:06.183 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:13:30, start=2025-10-27T11:10:00, end=2025-10-27T11:12:30, updates=218, granule=120000ms, delay=210000ms 2025-10-27 11:16:06.189 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:13:30, target=2025-10-27T11:12:30, granule=120000ms, delay=60000ms) 2025-10-27 11:16:06.189 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:12:30, target=2025-10-27T11:02:30, granule=600000ms, delay=600000ms) 2025-10-27 11:16:06.190 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=186ms, period=120000ms 2025-10-27 11:16:06.248 INFO 2458 --- [ocp-schedules-29,440c282be1b90997,9354c2b4f2fb28ce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:06.540 INFO 2458 --- [ocp-async-2,0bc47fbc31ef2a17,2aeee7a205c82574] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:06.540 WARN 2458 --- [ocp-async-3,0447a5c35c6135f2,c6bb3984878935ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:07.013 INFO 2458 --- [ocp-schedules-28,217cd3eaad12029b,3418da16f4b006bd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:16:07.021 INFO 2458 --- [ocp-schedules-28,217cd3eaad12029b,3418da16f4b006bd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:07.021 INFO 2458 --- [ocp-schedules-28,217cd3eaad12029b,3418da16f4b006bd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:07.857 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:16:07.857+08:00, tenantSize=6 2025-10-27 11:16:07.887 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761534967857000 2025-10-27 11:16:07.887 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761534667824000 2025-10-27 11:16:07.919 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:16:07.930 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761534967924000 2025-10-27 11:16:07.930 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761534667736000 2025-10-27 11:16:07.969 INFO 2458 --- [ocp-schedules-27,4b94614323ba7dfe,6e9ed94d2651e26a] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:16:08.517 INFO 2458 --- [ocp-async-2,4623694a959fc614,ef67e0e08ce315ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:08.540 INFO 2458 --- [ocp-async-10,3fae9ef6748667a3,059f93f4785e1ee0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:08.540 WARN 2458 --- [ocp-async-7,acdde383b591d1c2,4b2c59f4c6b6f7a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:08.707 INFO 2458 --- [prometheus-minute-schedule-1,4b8612ef91576fe9,b1d0e1abff35bd0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:08.714 INFO 2458 --- [prometheus-minute-schedule-1,4b8612ef91576fe9,b1d0e1abff35bd0d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:09.177 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:09.179 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:10.012 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:16:10.012 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:16:10.046 INFO 2458 --- [Check obproxy1,d0c0ff59f38f0498,a0d41c00b1df9db9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:16:10.046 INFO 2458 --- [Check obproxy1,d0c0ff59f38f0498,a0d41c00b1df9db9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:16:10.050 INFO 2458 --- [Collect obproxy1,d1bed7db194b0983,a9bbd34e4e90db23] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:16:10.050 INFO 2458 --- [Collect obproxy1,d1bed7db194b0983,a9bbd34e4e90db23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:16:10.076 INFO 2458 --- [Check obproxy1,d0c0ff59f38f0498,a0d41c00b1df9db9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:16:10.079 INFO 2458 --- [Collect obproxy1,d1bed7db194b0983,a9bbd34e4e90db23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:16:10.079 INFO 2458 --- [Check obproxy1,d0c0ff59f38f0498,a0d41c00b1df9db9] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:16:10.079 INFO 2458 --- [Check obproxy1,d0c0ff59f38f0498,a0d41c00b1df9db9] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:16:10.081 INFO 2458 --- [Collect obproxy1,d1bed7db194b0983,a9bbd34e4e90db23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:16:10.084 INFO 2458 --- [Collect obproxy1,d1bed7db194b0983,a9bbd34e4e90db23] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:16:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:16:10.540 INFO 2458 --- [ocp-async-10,d0d5e23f9862ad16,a47e3680d584c167] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:10.540 WARN 2458 --- [ocp-async-5,2e22066a59466483,5fe0787301641c1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:10.943 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534900, nodeCount=194, affectRows=194 2025-10-27 11:16:11.707 INFO 2458 --- [prometheus-minute-schedule-1,2204a693fb53a9b0,6e32303f7f2d2fcd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:11.714 INFO 2458 --- [prometheus-minute-schedule-1,2204a693fb53a9b0,6e32303f7f2d2fcd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:11.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:11.717457+08:00, lastProcessTime=2025-10-27T11:16:01.713991+08:00 2025-10-27 11:16:11.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:11.717457+08:00 2025-10-27 11:16:12.179 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:12.180 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:12.540 INFO 2458 --- [ocp-async-10,02f10fd74a35ea88,58cb68794bd09add] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:12.541 WARN 2458 --- [ocp-async-1,ce67a22dc9eb2971,fcdcaa08e9052477] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:13.435 INFO 2458 --- [ocp-schedules-8,e91ba4b0c69906b6,78a1ed1306f3b8c7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:13.454 WARN 2458 --- [scanner-112160205-1,e91ba4b0c69906b6,7e4f65b5f6e41334] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:13.543 INFO 2458 --- [alarm-detect-6,e91ba4b0c69906b6,6e7c4fb3d2631949] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534945 ~ 1761534963 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534960, value=41.253)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534960, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534960=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:16:13.847 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534900, nodeCount=623, affectRows=623 2025-10-27 11:16:14.225 INFO 2458 --- [ocp-schedules-27,f7b8aa9c448c9c98,465b6fb3d17ce33c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:14.228 INFO 2458 --- [ocp-schedules-27,f7b8aa9c448c9c98,465b6fb3d17ce33c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:14.541 INFO 2458 --- [ocp-async-10,552b8f181550c006,87a863b6c862e0ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:14.541 WARN 2458 --- [ocp-async-6,d94cb93e3c104b72,642e3921da084e94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:14.707 INFO 2458 --- [prometheus-minute-schedule-1,ec54017b981291bd,87247c7c55f9d270] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:14.714 INFO 2458 --- [prometheus-minute-schedule-1,ec54017b981291bd,87247c7c55f9d270] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:14.735 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=1000, affectRows=1000 2025-10-27 11:16:14.739 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=994, validCount=992 2025-10-27 11:16:14.864 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=1000, affectRows=1000 2025-10-27 11:16:14.868 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=999, validCount=995 2025-10-27 11:16:14.897 ERROR 2458 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570101) Duplicate entry '835370602979598336-1761534840' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570101) Duplicate entry '835370602979598336-1761534840' for key 'PRIMARY' 2025-10-27 11:16:14.902 INFO 2458 --- [ocp-schedules-7,d8aba0a578f00eb1,ee0bd4b28be9e0bf] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:16:14.902+08:00, clusters=[1, 2] 2025-10-27 11:16:14.904 INFO 2458 --- [ocp-schedules-7,d8aba0a578f00eb1,ee0bd4b28be9e0bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:16:14.908 INFO 2458 --- [ocp-schedules-7,d8aba0a578f00eb1,ee0bd4b28be9e0bf] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:16:15.180 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:15.181 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:15.546 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=1000, affectRows=1000 2025-10-27 11:16:15.550 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=1000, validCount=996 2025-10-27 11:16:15.556 INFO 2458 --- [ocp-schedules-5,07970a65005208d1,62758fa9f622153d] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:16:15.556 INFO 2458 --- [ocp-schedules-5,07970a65005208d1,62758fa9f622153d] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:16:15.601 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=98, affectRows=98 2025-10-27 11:16:15.602 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=97, validCount=97 2025-10-27 11:16:15.617 INFO 2458 --- [Collect all tenants major compactions1,07970a65005208d1,089bfd8ab86a299a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:16:15.617 INFO 2458 --- [Collect all tenants major compactions2,07970a65005208d1,3f1b26bf9bb48ec0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:16:15.617 INFO 2458 --- [Collect all tenants major compactions3,07970a65005208d1,ee238d728b8f0130] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:16:15.618 INFO 2458 --- [Collect all tenants major compactions4,07970a65005208d1,cd4ab987a7e0cb29] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:16:15.619 INFO 2458 --- [Collect all tenants major compactions5,07970a65005208d1,5899d89573cd85fa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:16:15.619 INFO 2458 --- [Collect all tenants major compactions6,07970a65005208d1,7ff7d479da3f509d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:16:15.666 INFO 2458 --- [Collect all tenants major compactions6,07970a65005208d1,7ff7d479da3f509d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.667 INFO 2458 --- [Collect all tenants major compactions5,07970a65005208d1,5899d89573cd85fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.668 INFO 2458 --- [Collect all tenants major compactions6,07970a65005208d1,7ff7d479da3f509d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:16:15.668 INFO 2458 --- [Collect all tenants major compactions5,07970a65005208d1,5899d89573cd85fa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:16:15.671 INFO 2458 --- [Collect all tenants major compactions6,07970a65005208d1,7ff7d479da3f509d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:16:15.672 INFO 2458 --- [Collect all tenants major compactions5,07970a65005208d1,5899d89573cd85fa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:16:15.688 INFO 2458 --- [Collect all tenants major compactions3,07970a65005208d1,ee238d728b8f0130] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.690 INFO 2458 --- [Collect all tenants major compactions3,07970a65005208d1,ee238d728b8f0130] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:16:15.693 INFO 2458 --- [Collect all tenants major compactions3,07970a65005208d1,ee238d728b8f0130] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:16:15.701 INFO 2458 --- [Collect all tenants major compactions2,07970a65005208d1,3f1b26bf9bb48ec0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.703 INFO 2458 --- [Collect all tenants major compactions2,07970a65005208d1,3f1b26bf9bb48ec0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:16:15.703 INFO 2458 --- [Collect all tenants major compactions4,07970a65005208d1,cd4ab987a7e0cb29] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.705 INFO 2458 --- [Collect all tenants major compactions4,07970a65005208d1,cd4ab987a7e0cb29] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:16:15.706 INFO 2458 --- [Collect all tenants major compactions2,07970a65005208d1,3f1b26bf9bb48ec0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:16:15.709 INFO 2458 --- [Collect all tenants major compactions4,07970a65005208d1,cd4ab987a7e0cb29] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:16:15.711 INFO 2458 --- [Collect all tenants major compactions1,07970a65005208d1,089bfd8ab86a299a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:15.712 INFO 2458 --- [Collect all tenants major compactions1,07970a65005208d1,089bfd8ab86a299a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:16:15.715 INFO 2458 --- [Collect all tenants major compactions1,07970a65005208d1,089bfd8ab86a299a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:16:15.716 INFO 2458 --- [ocp-schedules-5,07970a65005208d1,62758fa9f622153d] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:16:15.719 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=1000, affectRows=1000 2025-10-27 11:16:15.723 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=996, validCount=992 2025-10-27 11:16:16.176 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162} 2025-10-27 11:16:16.187 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterV 2025-10-27 11:16:16.188 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.193 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 7 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.233 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:16:16.233 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.254 INFO 2458 --- [ocp-schedules-19,da14dd77c4b63f95,d9bb1600ffb719cb] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:16.353 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576353000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=4A1EE035FCA9161B451248C53FDD87A7, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.353Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.366 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576366000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=855A8CE018BD24C202E9C0577A67C539, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.366Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.368 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576368000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.368Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.368 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576368000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=AEE1D864C114B40CA73E5A66FADF96F1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.368Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.368 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576368000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=6B7DF153607C7CC6C2FE0CCC997ED614, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.368Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.374 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 9 2025-10-27 11:16:16.374 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.378 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:16:16.378 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.379 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576379000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=0283C73D519DB2A02CD44C6D5CC58541, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.379Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.382 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 4 2025-10-27 11:16:16.382 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.384 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:16:16.384 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.389 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 8 2025-10-27 11:16:16.389 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.391 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576391000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=F9494FF57B2C1C815D5EF9BEDA853037, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.391Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.404 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:16:16.404 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.414 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 103 2025-10-27 11:16:16.414 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.416 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.416 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=AEE1D864C114B40CA73E5A66FADF96F1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.417 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.417 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=4A1EE035FCA9161B451248C53FDD87A7, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.418 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.418 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=6B7DF153607C7CC6C2FE0CCC997ED614, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.419 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.419 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=0283C73D519DB2A02CD44C6D5CC58541, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.424 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.424 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.425 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.425 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.425 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.425 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.427 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.427 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.428 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.428 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.430 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.430 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.431 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.431 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=F9494FF57B2C1C815D5EF9BEDA853037, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.436 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.436 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.452 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.452 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=855A8CE018BD24C202E9C0577A67C539, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=4A1EE035FCA9161B451248C53FDD87A7, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.460 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.462 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.462 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=6B7DF153607C7CC6C2FE0CCC997ED614, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.462 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.462 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=AEE1D864C114B40CA73E5A66FADF96F1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.462 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.463 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=0283C73D519DB2A02CD44C6D5CC58541, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.465 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:16:16.465 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.368, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.465 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:16:16.467 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:16:16.467 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.368, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=6B7DF153607C7CC6C2FE0CCC997ED614, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.468 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:16:16.469 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:16:16.477 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.477 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=F9494FF57B2C1C815D5EF9BEDA853037, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.480 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:16:16.481 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.391, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=F9494FF57B2C1C815D5EF9BEDA853037, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.495 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:16.495 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=855A8CE018BD24C202E9C0577A67C539, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:16.497 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.353, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=4A1EE035FCA9161B451248C53FDD87A7, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.500 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:16:16.541 INFO 2458 --- [ocp-async-10,e85a6221cd6b7175,fb99d9a54ddf8e80] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:16.541 WARN 2458 --- [ocp-async-4,deef96ced4292a7f,d2afaec11d90b081] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:16.557 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.379, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=0283C73D519DB2A02CD44C6D5CC58541, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.602 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.366, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=855A8CE018BD24C202E9C0577A67C539, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.843 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.368, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=6, sqlId=AEE1D864C114B40CA73E5A66FADF96F1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:16.844 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.845 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.845 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:16:16.845 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.848 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.849 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.849 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:16:16.849 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.852 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.853 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.853 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:16:16.853 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.856 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.857 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.857 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:16:16.857 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.859 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:16:16.866 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:12:15.73, end=2025-10-27T11:14:16.162},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:16:16.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:16:16.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:16:16.986 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576986000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.986Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:16.994 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:16:16.994 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:16.997 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448576997000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=6FB0409CBBA181A7F58484E25CAF92D9, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:16:16.997Z, endTime=2025-10-27T03:14:16.162Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:16:17.005 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:16:17.005 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:16:17.024 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:17.024 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:17.030 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:17.030 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:17.030 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:17.030 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=6FB0409CBBA181A7F58484E25CAF92D9, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:17.035 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:16:17.035 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:16:17.056 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:17.056 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:17.057 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:16:17.058 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534856162000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=6FB0409CBBA181A7F58484E25CAF92D9, sqlIdList=null, dynamicSql=false) 2025-10-27 11:16:17.061 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:16:17.061 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.986, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:17.062 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:16:17.062 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:16:16.997, end=2025-10-27T11:14:16.162}, clusterId=2, tenantId=5, sqlId=6FB0409CBBA181A7F58484E25CAF92D9, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:16:17.062 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:16:17.707 INFO 2458 --- [prometheus-minute-schedule-1,6e927a789204cc6a,c120db94e8e70c38] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:17.716 INFO 2458 --- [prometheus-minute-schedule-1,6e927a789204cc6a,c120db94e8e70c38] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:17.909 INFO 2458 --- [ocp-schedules-11,b6791e8567b5ff6e,6b9fdeb6d64d0ada] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:16:17.909 INFO 2458 --- [ocp-schedules-11,b6791e8567b5ff6e,6b9fdeb6d64d0ada] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:16:17.918 INFO 2458 --- [ocp-schedules-11,b6791e8567b5ff6e,6b9fdeb6d64d0ada] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:16:17.918 INFO 2458 --- [ocp-schedules-11,b6791e8567b5ff6e,6b9fdeb6d64d0ada] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:16:18.134 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.144 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=n 2025-10-27 11:16:18.153 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.160 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.161 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.163 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.168 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.174 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.177 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.181 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:18.181 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:18.181 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.183 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.184 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.185 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.186 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.187 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.188 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.190 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.191 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.192 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.193 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.194 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.195 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.198 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.201 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.203 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.206 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:10:17.579, end=2025-10-27T11:15:18.018} 2025-10-27 11:16:18.206 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:16:18.517 INFO 2458 --- [ocp-async-10,9a7736448de3f9eb,5251a629eec4907e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:18.541 INFO 2458 --- [ocp-async-9,b70af97363d2c242,c231832eee178c98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:18.541 WARN 2458 --- [ocp-async-7,da75d27cd99aa5d9,3a0fc10cd8e08a30] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:19.699 INFO 2458 --- [ocp-schedules-24,cbd043efcb24f5a9,a32241adbcf03776] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:16:20.541 WARN 2458 --- [ocp-async-2,eaffbbc4ba4a738a,10b375a67b5e924f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:20.541 INFO 2458 --- [ocp-async-8,2e952e81e2fa85f3,9f6c37bb105a92de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:20.625 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bf3b83e5c60cfa0a,495236b915b7d93f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=bf3b83e5c60cfa0a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:16:20.625 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bf3b83e5c60cfa0a,495236b915b7d93f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=bf3b83e5c60cfa0a, duration=1 ms] 2025-10-27 11:16:20.707 INFO 2458 --- [prometheus-minute-schedule-1,51c7397bb4cec9a6,fc56998a0d67176b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:20.716 INFO 2458 --- [prometheus-minute-schedule-1,51c7397bb4cec9a6,fc56998a0d67176b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:21.059 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534900, nodeCount=1000, affectRows=1000 2025-10-27 11:16:21.063 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534900, nodeCount=996, validCount=984 2025-10-27 11:16:21.181 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:21.182 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:21.409 INFO 2458 --- [ocp-schedules-13,355b6b974c7e455b,37843fab2634f482] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:21.409 INFO 2458 --- [ocp-schedules-25,4d48b40d3d527df2,ee288f919064e9c8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:21.409 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:16:21.410 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:16:21.410 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:16:21.410 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:16:21.410 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:16:21.410 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:16:21.411 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:16:21.411 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:16:21.411 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:16:21.411 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:16:21.423 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:21.424 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:21.424 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:21.424 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:21.425 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:21.425 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:21.425 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:21.428 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:21.428 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:21.429 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:21.429 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:21.429 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:21.429 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:21.429 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:21.430 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:21.430 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:21.431 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:21.431 INFO 2458 --- [ocp-schedules-12,86847fcd100b1755,60411cfa8b7b5a85] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:21.431 INFO 2458 --- [host-check-clockdiff1,86847fcd100b1755,a5c5e99ee51897a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:16:21.432 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:21.432 INFO 2458 --- [host-check-clockdiff3,86847fcd100b1755,123d0dfc9b5fce6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:16:21.433 INFO 2458 --- [host-check-clockdiff2,86847fcd100b1755,818b4d035b380db3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:16:21.433 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:21.434 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:21.436 INFO 2458 --- [host-check-clockdiff4,86847fcd100b1755,4ab2002e8bfc669f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:16:21.453 INFO 2458 --- [host-check-clockdiff5,86847fcd100b1755,25a22058e75685b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:16:21.453 INFO 2458 --- [ocp-schedules-23,99646214e7e620a1,c50bb1961bab3787] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:16:21.479 INFO 2458 --- [host-check-clockdiff1,86847fcd100b1755,a5c5e99ee51897a6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534981 0 0 , error= 2025-10-27 11:16:21.479 INFO 2458 --- [host-check-clockdiff4,86847fcd100b1755,4ab2002e8bfc669f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534981 0 0 , error= 2025-10-27 11:16:21.479 INFO 2458 --- [host-check-clockdiff2,86847fcd100b1755,818b4d035b380db3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534981 -1 -1 , error= 2025-10-27 11:16:21.484 INFO 2458 --- [host-check-clockdiff3,86847fcd100b1755,123d0dfc9b5fce6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534981 -1 -1 , error= 2025-10-27 11:16:21.489 INFO 2458 --- [host-check-clockdiff5,86847fcd100b1755,25a22058e75685b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761534981 0 0 , error= 2025-10-27 11:16:21.512 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:16:21.516 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:16:21.517 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:16:21.518 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:16:21.517498382+08:00 2025-10-27 11:16:21.519 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:16:21.519+08:00 2025-10-27 11:16:21.519 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:16:21.519 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:16:21.519 INFO 2458 --- [host-check-agent-status1,355b6b974c7e455b,dda5a82742ca8ede] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:16:21.521 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:16:21.521 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:16:21.521 INFO 2458 --- [host-check-agent-status5,355b6b974c7e455b,682eb44d7c639cfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:16:21.529 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:16:21.529 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:16:21.530 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:16:21.530056923+08:00 2025-10-27 11:16:21.530 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:16:21.519+08:00 and status AVAILABLE 2025-10-27 11:16:21.531 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:16:21.530+08:00 2025-10-27 11:16:21.531 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:16:21.531 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:16:21.530714715+08:00 2025-10-27 11:16:21.531 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:16:21.531841259+08:00 2025-10-27 11:16:21.532 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:16:21.532 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:16:21.531+08:00 2025-10-27 11:16:21.534 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:16:21.534 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:16:21.531+08:00 2025-10-27 11:16:21.536 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:16:21.536 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:16:21.535922970+08:00 2025-10-27 11:16:21.537 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:16:21.536+08:00 2025-10-27 11:16:21.537 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:16:21.539 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:16:21.539 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:16:21.539 INFO 2458 --- [host-check-agent-status4,355b6b974c7e455b,36c568cdbfc11403] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:16:21.540 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:16:21.530+08:00 and status AVAILABLE 2025-10-27 11:16:21.540 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:16:21.542 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:16:21.542 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:16:21.542 INFO 2458 --- [host-check-agent-status3,355b6b974c7e455b,dd4aefd9ac0be2ca] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:16:21.543 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:16:21.543 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:16:21.543 INFO 2458 --- [host-check-agent-status2,355b6b974c7e455b,a3045bb61ff788e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:16:21.545 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:16:21.531+08:00 and status AVAILABLE 2025-10-27 11:16:21.545 INFO 2458 --- [ocp-schedules-19,fab55bad42ab82ee,8e1698e0143e6ae9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:16:21.546 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:16:21.536+08:00 and status AVAILABLE 2025-10-27 11:16:21.546 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:16:21.531+08:00 and status AVAILABLE 2025-10-27 11:16:21.547 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:16:21.549 INFO 2458 --- [ocp-schedules-3,a54d8be4f2ee03c6,12cee9e92e118ba2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:16:21.550 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:16:21.550 INFO 2458 --- [ocp-schedules-19,fab55bad42ab82ee,8e1698e0143e6ae9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:21.552 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:16:21.552 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:16:21.554 INFO 2458 --- [ocp-schedules-6,ccc42ebe5982e697,f522f04d66bbfbac] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:16:21.555 INFO 2458 --- [ocp-schedules-3,a54d8be4f2ee03c6,12cee9e92e118ba2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:21.559 INFO 2458 --- [ocp-schedules-6,ccc42ebe5982e697,f522f04d66bbfbac] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:21.560 INFO 2458 --- [ocp-schedules-3,a54d8be4f2ee03c6,12cee9e92e118ba2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:16:21.565 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:21.567 INFO 2458 --- [ocp-schedules-30,16b513c87e4c2a30,51fc08f774244b49] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:16:21.570 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:21.577 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:21.579 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:21.580 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:21.580 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:21.581 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:21.584 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:21.584 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:21.584 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:21.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:21.602 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:16:21.623 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:16:21.628 INFO 2458 --- [ocp-schedules-19,fab55bad42ab82ee,8e1698e0143e6ae9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:16:21.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:21.648 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:21.649 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:21.655 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:21.658 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:21.659 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:21.665 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:16:21.668 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:16:21.670 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.670 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:21.675 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:16:21.677 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:16:21.692 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:16:21.709 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:16:21.715 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:16:21.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:21.717634+08:00, lastProcessTime=2025-10-27T11:16:11.717457+08:00 2025-10-27 11:16:21.719 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:21.720 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.721 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:16:21.725 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:16:21.725 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:16:21.727 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:16:21.763 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:21.764 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.765 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:16:21.766 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:21.769 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:21.779 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:21.780 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:21.785 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:16:21.786 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:21.790 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:16:21.791 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:21.792 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.792 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:21.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.796 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:16:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:16:21.799 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:21.807 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:16:21.812 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:16:21.816 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:16:21.819 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:21.819 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:21.820 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:16:21.825 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:16:21.838 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:16:21.841 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:16:21.850 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:21.852 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:21.860 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:21.861 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.861 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:21.861 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.866 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:16:21.866 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:21.867 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:21.870 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:16:21.876 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:21.878 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:16:21.882 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:16:21.883 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:16:21.892 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:16:21.896 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:16:21.908 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:21.913 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:21.915 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:16:21.923 INFO 2458 --- [host-check-host-status4,4d48b40d3d527df2,4cb0945aa4aabe83] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:16:21.935 INFO 2458 --- [host-check-host-status2,4d48b40d3d527df2,f2cbc12706c9eca9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:16:21.943 INFO 2458 --- [host-check-host-status3,4d48b40d3d527df2,1ab87167d5eea8eb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:16:21.948 INFO 2458 --- [host-check-host-status5,4d48b40d3d527df2,88a0a7ad6c1c07ad] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:16:21.955 INFO 2458 --- [host-check-host-status1,4d48b40d3d527df2,70118f55bbfa7f15] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:16:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:16:22.044 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:16:22.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.045 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:16:22.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:16:22.069 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:16:22.093 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:22.093 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.094 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:16:22.115 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:16:22.115 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:16:22.186 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:22.186 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:22.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:22.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:22.324 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:21.717634+08:00 2025-10-27 11:16:22.336 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:22.382 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:22.382 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.383 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:22.387 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:16:22.399 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:16:22.413 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:22.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:22.470 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:16:22.541 INFO 2458 --- [ocp-async-5,23a3d980d11de23b,2a985ca7fafecd2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:22.541 WARN 2458 --- [ocp-async-3,db9dc73fa5a21b91,8886e1a821bd67cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:22.546 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:16:22.547 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.547 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:16:22.549 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.550 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:16:22.592 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:16:22.632 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:22.634 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:22.634 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:16:22.702 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:16:22.702 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:16:22.702 INFO 2458 --- [ocp-schedules-6,ccc42ebe5982e697,f522f04d66bbfbac] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:16:23.549 INFO 2458 --- [ocp-schedules-7,7b94ee62a3dfcd02,463380772df08b2b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:23.558 WARN 2458 --- [scanner-112160362-1,7b94ee62a3dfcd02,285df0eba2b53100] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:23.609 INFO 2458 --- [alarm-detect-1,7b94ee62a3dfcd02,9b5a7f5c1094420a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:23.613 INFO 2458 --- [alarm-detect-1,7b94ee62a3dfcd02,9b5a7f5c1094420a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:23.642 INFO 2458 --- [alarm-detect-16,7b94ee62a3dfcd02,409e93d1f0c16dbf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:23.644 INFO 2458 --- [alarm-detect-4,7b94ee62a3dfcd02,98a48c245dfab2c3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:23.648 INFO 2458 --- [alarm-detect-16,7b94ee62a3dfcd02,409e93d1f0c16dbf] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:23.649 INFO 2458 --- [alarm-detect-4,7b94ee62a3dfcd02,98a48c245dfab2c3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:23.709 INFO 2458 --- [prometheus-minute-schedule-1,1d4871fafda644c9,c14c51928e321092] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:23.716 INFO 2458 --- [prometheus-minute-schedule-1,1d4871fafda644c9,c14c51928e321092] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:23.750 INFO 2458 --- [alarm-detect-2,7b94ee62a3dfcd02,6894aeb07e46f496] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:23.754 INFO 2458 --- [alarm-detect-2,7b94ee62a3dfcd02,6894aeb07e46f496] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:24.182 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:24.183 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:24.289 INFO 2458 --- [ocp-schedules-4,2e4d6a4725ecd58d,ef5a1ac1dbe2279e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:24.293 INFO 2458 --- [ocp-schedules-4,2e4d6a4725ecd58d,ef5a1ac1dbe2279e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:24.541 WARN 2458 --- [ocp-async-7,3cc2fcc253812164,e4c634c395799006] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:24.541 INFO 2458 --- [ocp-async-7,a445177990245a03,0f269035c5cd6834] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:26.260 INFO 2458 --- [ocp-schedules-2,62fb1c2a2877ab8b,1d71faf6d4b68368] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:26.541 INFO 2458 --- [ocp-async-6,3edc12785c8ca17b,f31359d7edc8d36e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:26.542 WARN 2458 --- [ocp-async-4,53b1ef45773396e9,15652c47a5844e71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:26.707 INFO 2458 --- [prometheus-minute-schedule-1,ba3ef910779b4310,20d7ff38f7a36fa7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:26.716 INFO 2458 --- [prometheus-minute-schedule-1,ba3ef910779b4310,20d7ff38f7a36fa7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:27.024 INFO 2458 --- [ocp-schedules-18,278b87fbd509b76d,7ca0399577dd3678] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:16:27.032 INFO 2458 --- [ocp-schedules-18,278b87fbd509b76d,7ca0399577dd3678] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:27.032 INFO 2458 --- [ocp-schedules-18,278b87fbd509b76d,7ca0399577dd3678] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:27.183 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:27.184 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:28.518 INFO 2458 --- [ocp-async-9,0e0845ac3a6d3d37,6af99de8f6a64cc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:28.542 INFO 2458 --- [ocp-async-8,0e631a950968c743,333707aa48466d70] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:28.542 WARN 2458 --- [ocp-async-5,ddc7c817d34a3956,b1509786819560d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:28.918 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:16:28.918 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:16:28.920 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:16:28.920 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:16:28.922 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:16:28.922 INFO 2458 --- [ocp-schedules-2,a8adee755b2386bc,b7e9348a7c7325fe] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:16:29.376 INFO 2458 --- [ocp-schedules-9,cd0b80c5298b644a,7dd694b50077e1d1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:16:29.386 INFO 2458 --- [ocp-schedules-9,cd0b80c5298b644a,7dd694b50077e1d1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:16:29.386 INFO 2458 --- [ocp-schedules-9,cd0b80c5298b644a,7dd694b50077e1d1] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:16:29.707 INFO 2458 --- [prometheus-minute-schedule-1,20022bf2137fcf3e,b76bcd7b1be507e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:29.717 INFO 2458 --- [prometheus-minute-schedule-1,20022bf2137fcf3e,b76bcd7b1be507e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:30.184 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:30.185 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:30.542 INFO 2458 --- [ocp-async-1,a6392ac5b9d6f895,cc8414047c7d6ada] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:30.542 WARN 2458 --- [ocp-async-7,20bb1bdd37080336,576196d890bfe003] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:31.715868+08:00, lastProcessTime=2025-10-27T11:16:21.717634+08:00 2025-10-27 11:16:31.948 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:31.715868+08:00 2025-10-27 11:16:32.542 WARN 2458 --- [ocp-async-4,747e71204f73bd91,8136a27660bea616] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:32.542 INFO 2458 --- [ocp-async-6,71e492cade3b477d,c240570a7381d570] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:32.707 INFO 2458 --- [prometheus-minute-schedule-1,eb90f49acd50c442,d51c3e6e41292cdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:32.715 INFO 2458 --- [prometheus-minute-schedule-1,eb90f49acd50c442,d51c3e6e41292cdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:33.185 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:33.186 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:33.792 INFO 2458 --- [ocp-schedules-19,9449afe32ce852b5,83aef672b63563a5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:33.810 WARN 2458 --- [scanner-112160627-1,9449afe32ce852b5,a35b0a7e3b5585aa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:33.814 INFO 2458 --- [ocp-schedules-6,60b39f5b3e63b2f2,1494495d057b3f10] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:16:33.814 INFO 2458 --- [ocp-schedules-6,60b39f5b3e63b2f2,1494495d057b3f10] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:16:33.934 INFO 2458 --- [alarm-detect-2,9449afe32ce852b5,3fb813193da63f0b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534960 ~ 1761534983 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534975, value=35.174)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534975, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534975=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:16:34.300 INFO 2458 --- [ocp-schedules-7,0655378cbe8148bb,b5807254c2cfb9e0] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:34.304 INFO 2458 --- [ocp-schedules-7,0655378cbe8148bb,b5807254c2cfb9e0] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:34.542 INFO 2458 --- [ocp-async-9,45404e04804f1db0,436fb3a5c8439ee3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:34.542 WARN 2458 --- [ocp-async-3,fb7ab5691b7972b6,0c9053475dd53a2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:35.707 INFO 2458 --- [prometheus-minute-schedule-1,9ffaa571704bb495,e3069882ec85fc6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:35.714 INFO 2458 --- [prometheus-minute-schedule-1,9ffaa571704bb495,e3069882ec85fc6b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:36.186 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:36.187 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:36.269 INFO 2458 --- [ocp-schedules-30,af1e0445f852e9f2,37fcff5612905d0e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:36.542 WARN 2458 --- [ocp-async-7,056eb81810e96531,8b67471b5046cc57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:36.542 INFO 2458 --- [ocp-async-10,3e32fcfb6306324d,92c08d1d06e2f46b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:36.803 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:16:36.803 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:16:36.906 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:36.907 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:16:36.919 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:36.920 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:16:36.981 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:36.982 INFO 2458 --- [ocp-schedules-21,e55ff8b829033ce5,107f4569cbb1e418] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:16:37.580 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,91072fd377989506,aca41af46c9bc1b1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=91072fd377989506, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:16:37.580 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,91072fd377989506,aca41af46c9bc1b1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=91072fd377989506, duration=1 ms] 2025-10-27 11:16:37.708 INFO 2458 --- [prometheus-cache-monitor1,aabf6477e2688b7f,d47404432e373ff4] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20357,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:16:37.708 INFO 2458 --- [prometheus-cache-monitor1,aabf6477e2688b7f,d47404432e373ff4] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:16:38.100 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:38.100 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:16:38.101 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:16:38.101 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:16:38.101 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:16:38.101 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:16:38.114 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:38.114 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:38.114 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:38.115 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:38.115 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:38.118 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:38.118 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:38.118 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:38.118 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:38.119 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:38.177 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:16:38.183 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:16:38.183 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:16:38.183 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:16:38.184 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:16:38.185 INFO 2458 --- [ocp-schedules-6,e4bb3e4d77dabc98,f4e17021a6473738] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:16:38.518 INFO 2458 --- [ocp-async-1,6f3658e7a819f3dd,b293ea0269ef4d4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:38.542 WARN 2458 --- [ocp-async-2,96f06e36b8d99a4b,d8dff4309b49c6f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:38.542 INFO 2458 --- [ocp-async-6,426005cc0c200411,3d76f2e9c7ee6ffd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:38.663 INFO 2458 --- [ocp-schedules-22,540fdf3faa8d193a,bd3f743e04e48150] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:16:38.669 INFO 2458 --- [ocp-schedules-22,540fdf3faa8d193a,bd3f743e04e48150] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:38.669 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:16:38.669 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:16:38.669 INFO 2458 --- [ocp-schedules-22,540fdf3faa8d193a,bd3f743e04e48150] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:16:38.707 INFO 2458 --- [prometheus-minute-schedule-1,9d5dfbf28decae37,06d94ef02828a846] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:38.714 INFO 2458 --- [prometheus-minute-schedule-1,9d5dfbf28decae37,06d94ef02828a846] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:39.187 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:39.188 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:40.008 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:16:40.008 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:16:40.042 INFO 2458 --- [Check obproxy1,6097aeb638819bcc,4303a0e75b44838b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:16:40.042 INFO 2458 --- [Check obproxy1,6097aeb638819bcc,4303a0e75b44838b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:16:40.046 INFO 2458 --- [Collect obproxy1,9f2c64581b797b78,392c84a6c7e8d86c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:16:40.046 INFO 2458 --- [Collect obproxy1,9f2c64581b797b78,392c84a6c7e8d86c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:16:40.071 INFO 2458 --- [Check obproxy1,6097aeb638819bcc,4303a0e75b44838b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:16:40.074 INFO 2458 --- [Check obproxy1,6097aeb638819bcc,4303a0e75b44838b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:16:40.074 INFO 2458 --- [Check obproxy1,6097aeb638819bcc,4303a0e75b44838b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:16:40.074 INFO 2458 --- [Collect obproxy1,9f2c64581b797b78,392c84a6c7e8d86c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:16:40.077 INFO 2458 --- [Collect obproxy1,9f2c64581b797b78,392c84a6c7e8d86c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:16:40.081 INFO 2458 --- [Collect obproxy1,9f2c64581b797b78,392c84a6c7e8d86c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:16:40.542 WARN 2458 --- [ocp-async-5,552ffd30a9db9bf5,6fc333094c76abd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:40.542 INFO 2458 --- [ocp-async-5,2026fece0c284f5a,b8ff591216702552] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:40.951 INFO 2458 --- [ocp-schedules-27,053d9d5bba148da5,8a195d615ab15a9b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:16:40.952 INFO 2458 --- [ocp-schedules-27,053d9d5bba148da5,8a195d615ab15a9b] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:16:41.707 INFO 2458 --- [prometheus-minute-schedule-1,5fb82d16417c5833,49367db1d16d5cd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:41.714 INFO 2458 --- [prometheus-minute-schedule-1,5fb82d16417c5833,49367db1d16d5cd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:41.738 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:41.737367+08:00, lastProcessTime=2025-10-27T11:16:31.715868+08:00 2025-10-27 11:16:41.963 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:41.737367+08:00 2025-10-27 11:16:42.188 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:42.189 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:42.543 WARN 2458 --- [ocp-async-7,8be8e77840ecfef0,35f98690d946bf2f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:42.543 INFO 2458 --- [ocp-async-7,96b74350bd56722a,337f39db9b866d67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:43.951 INFO 2458 --- [ocp-schedules-11,31c234a29a465b46,5d140964981abff8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:44.033 INFO 2458 --- [alarm-detect-15,31c234a29a465b46,b6ffbf7fd3565deb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:44.037 INFO 2458 --- [alarm-detect-15,31c234a29a465b46,b6ffbf7fd3565deb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:16:44.059 WARN 2458 --- [scanner-112160873-1,31c234a29a465b46,51e7e34cc34e99ef] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:44.076 INFO 2458 --- [alarm-detect-17,31c234a29a465b46,b11756497886ebcc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:44.077 INFO 2458 --- [ocp-schedules-14,3a7d83c6fca46a95,b7a31f2b93c4f051] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:16:44.077 INFO 2458 --- [alarm-detect-5,31c234a29a465b46,d9fbfc8ad444ead8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:44.081 INFO 2458 --- [alarm-detect-17,31c234a29a465b46,b11756497886ebcc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:16:44.081 INFO 2458 --- [alarm-detect-5,31c234a29a465b46,d9fbfc8ad444ead8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:16:44.197 INFO 2458 --- [alarm-detect-1,31c234a29a465b46,2e6c44eb1c45ed42] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:44.201 INFO 2458 --- [alarm-detect-1,31c234a29a465b46,2e6c44eb1c45ed42] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:16:44.242 INFO 2458 --- [alarm-detect-8,31c234a29a465b46,871e7b7288b6e1d3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534975 ~ 1761534994 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534990, value=37.664)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761534990, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761534990=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:16:44.312 INFO 2458 --- [ocp-schedules-30,1573ceb81049d984,bb9b30944a543106] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:44.314 INFO 2458 --- [ocp-schedules-30,1573ceb81049d984,bb9b30944a543106] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:44.543 WARN 2458 --- [ocp-async-1,ca9962ad1fe7969b,6cdcaff7c3d21060] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:44.543 INFO 2458 --- [ocp-async-2,5f894920e27b65ec,7b7a9d44d0c99d30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:44.707 INFO 2458 --- [prometheus-minute-schedule-1,32662b01fbc252fc,7ba0e0a451817c7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:44.714 INFO 2458 --- [prometheus-minute-schedule-1,32662b01fbc252fc,7ba0e0a451817c7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:45.102 INFO 2458 --- [prometheus-second-schedule-1,e28a9c5e4c476ac1,2abaf73bef355f1e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:16:45.189 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:45.190 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:46.279 INFO 2458 --- [ocp-schedules-29,2592ee7b1afccdca,11d606d6f33e8882] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:46.543 WARN 2458 --- [ocp-async-6,c403d54efe689922,93018c0a5345a446] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:46.543 INFO 2458 --- [ocp-async-9,1c25b200693ef7f6,084e010254408ed3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:47.035 INFO 2458 --- [ocp-schedules-4,726e5b283df93a6e,89f8cbd704f4682f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:16:47.044 INFO 2458 --- [ocp-schedules-4,726e5b283df93a6e,89f8cbd704f4682f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:47.044 INFO 2458 --- [ocp-schedules-4,726e5b283df93a6e,89f8cbd704f4682f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:16:47.707 INFO 2458 --- [prometheus-minute-schedule-1,68ba3baf27b2c775,77fa02e88e131a91] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:47.715 INFO 2458 --- [prometheus-minute-schedule-1,68ba3baf27b2c775,77fa02e88e131a91] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:48.062 INFO 2458 --- [ocp-schedules-18,89a59fad9a2a1023,292056200e17ce72] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:16:48.062 INFO 2458 --- [ocp-schedules-18,89a59fad9a2a1023,292056200e17ce72] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:16:48.070 INFO 2458 --- [ocp-schedules-18,89a59fad9a2a1023,292056200e17ce72] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:16:48.070 INFO 2458 --- [ocp-schedules-18,89a59fad9a2a1023,292056200e17ce72] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:16:48.190 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:48.191 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:48.518 INFO 2458 --- [ocp-async-9,26596e805c1f61d1,c267c533de521df6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:48.543 INFO 2458 --- [ocp-async-8,1055656bdfa790a4,9f75ac086fc86ecf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:48.543 WARN 2458 --- [ocp-async-7,d57b17cc9aea159e,3e38126f8f1b501c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:48.545 INFO 2458 --- [ocp-schedules-6,b0e1ad58974603b7,a9a4419d1a7da93e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:16:48.545+08:00, clusters=[1, 2] 2025-10-27 11:16:48.559 INFO 2458 --- [ocp-schedules-6,b0e1ad58974603b7,a9a4419d1a7da93e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:16:48.562 INFO 2458 --- [ocp-schedules-6,b0e1ad58974603b7,a9a4419d1a7da93e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:16:50.543 INFO 2458 --- [ocp-async-4,0560993c32ba4205,e1b86f0d8d1ed837] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:50.543 WARN 2458 --- [ocp-async-2,e1af53c35a336885,9d2f51dd3667d88a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:50.707 INFO 2458 --- [prometheus-minute-schedule-1,02af2043674be06f,b6e19dbb39d2ca7e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:50.714 INFO 2458 --- [prometheus-minute-schedule-1,02af2043674be06f,b6e19dbb39d2ca7e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:51.191 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:51.192 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:51.410 INFO 2458 --- [ocp-schedules-26,05d71465d74d963f,261029067e1f78ad] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:51.410 INFO 2458 --- [ocp-schedules-23,59fdae59ab8b35e2,129bea0f2d2cdcd9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:51.410 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:16:51.410 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:16:51.410 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:16:51.411 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:16:51.412 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:16:51.428 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:51.428 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:51.428 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:51.429 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:51.429 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:51.429 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:51.430 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:51.430 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:51.433 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:51.433 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:51.433 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:51.433 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:51.434 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:51.434 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:51.434 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:51.434 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:51.435 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:51.435 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:51.437 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:51.438 INFO 2458 --- [ocp-schedules-16,2a07bdf8663b177c,8b28cc1c2e932954] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:16:51.438 INFO 2458 --- [host-check-clockdiff1,2a07bdf8663b177c,42dfc7a623aa049a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:16:51.439 INFO 2458 --- [host-check-clockdiff2,2a07bdf8663b177c,92b570a3d46c33be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:16:51.439 INFO 2458 --- [host-check-clockdiff3,2a07bdf8663b177c,ecf0726d1e70034c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:16:51.439 INFO 2458 --- [host-check-clockdiff4,2a07bdf8663b177c,95093ce6550e61a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:16:51.439 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:51.439 INFO 2458 --- [host-check-clockdiff5,2a07bdf8663b177c,2bb45f87e84f7a4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:16:51.442 INFO 2458 --- [ocp-schedules-29,25b6196d7246b7cf,9e9fa28d6550a01a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:16:51.451 INFO 2458 --- [host-check-clockdiff1,2a07bdf8663b177c,42dfc7a623aa049a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535011 0 0 , error= 2025-10-27 11:16:51.471 INFO 2458 --- [host-check-clockdiff5,2a07bdf8663b177c,2bb45f87e84f7a4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535011 1 1 , error= 2025-10-27 11:16:51.481 INFO 2458 --- [host-check-clockdiff3,2a07bdf8663b177c,ecf0726d1e70034c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535011 -1 -1 , error= 2025-10-27 11:16:51.484 INFO 2458 --- [host-check-clockdiff2,2a07bdf8663b177c,92b570a3d46c33be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535011 -1 -1 , error= 2025-10-27 11:16:51.510 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:16:51.510 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:16:51.510 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:16:51.511 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:16:51.512 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:16:51.511320783+08:00 2025-10-27 11:16:51.512 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:16:51.512675535+08:00 2025-10-27 11:16:51.513 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:16:51.513 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:16:51.512+08:00 2025-10-27 11:16:51.513 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:16:51.512+08:00 2025-10-27 11:16:51.514 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:16:51.514 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:16:51.514 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:16:51.514 INFO 2458 --- [host-check-agent-status3,05d71465d74d963f,7d70fffddf8eb21d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:16:51.515 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:16:51.515 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:16:51.515 INFO 2458 --- [host-check-agent-status5,05d71465d74d963f,82be669cc1b10e41] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:16:51.518 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:16:51.518 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:16:51.518 INFO 2458 --- [host-check-agent-status1,05d71465d74d963f,c6d4604628be2cb3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:16:51.519 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:16:51.522 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:16:51.512+08:00 and status AVAILABLE 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-agent-status4,05d71465d74d963f,6413cb0632724558] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:16:51.512+08:00 and status AVAILABLE 2025-10-27 11:16:51.523 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:16:51.522920688+08:00 2025-10-27 11:16:51.524 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:16:51.523+08:00 2025-10-27 11:16:51.524 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:16:51.524 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:16:51.524 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:16:51.524 INFO 2458 --- [host-check-agent-status2,05d71465d74d963f,67282090c9bc6b97] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:16:51.525 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:16:51.524217150+08:00 2025-10-27 11:16:51.525 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:16:51.525+08:00 2025-10-27 11:16:51.527 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:16:51.530 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:16:51.531 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:16:51.530807735+08:00 2025-10-27 11:16:51.531 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:16:51.523+08:00 and status AVAILABLE 2025-10-27 11:16:51.531 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:16:51.531+08:00 2025-10-27 11:16:51.533 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:16:51.534 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:16:51.525+08:00 and status AVAILABLE 2025-10-27 11:16:51.535 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:16:51.538 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:16:51.539 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:16:51.531+08:00 and status AVAILABLE 2025-10-27 11:16:51.543 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:16:51.545 INFO 2458 --- [ocp-schedules-10,3b2973857083b4ce,4ad34ddab3c60d27] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:16:51.547 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:16:51.549 INFO 2458 --- [ocp-schedules-8,c32ecc3767a7ad31,5e17bf6cb5c19a04] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:16:51.550 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:16:51.551 INFO 2458 --- [ocp-schedules-10,3b2973857083b4ce,4ad34ddab3c60d27] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:51.552 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:51.554 INFO 2458 --- [ocp-schedules-18,fbfe83a8f180f86e,5cc7764f532bccdb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:16:51.554 INFO 2458 --- [ocp-schedules-8,c32ecc3767a7ad31,5e17bf6cb5c19a04] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:51.554 INFO 2458 --- [ocp-schedules-8,c32ecc3767a7ad31,5e17bf6cb5c19a04] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:16:51.555 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:16:51.555 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:51.557 INFO 2458 --- [ocp-schedules-18,fbfe83a8f180f86e,5cc7764f532bccdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:16:51.558 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:16:51.561 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:16:51.564 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:16:51.565 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:51.567 INFO 2458 --- [ocp-schedules-19,19e61d827e51dc8e,b3427f2aee6a1372] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:16:51.568 INFO 2458 --- [ocp-schedules-25,6f9761353b64941b,11a190b29005ec65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:16:51.568 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:51.569 INFO 2458 --- [ocp-schedules-1,5ed6d7dd57d4e116,f4f47b796c5410e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:16:51.577 INFO 2458 --- [ocp-schedules-25,6f9761353b64941b,11a190b29005ec65] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:16:51.580 INFO 2458 --- [ocp-schedules-1,5ed6d7dd57d4e116,f4f47b796c5410e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:16:51.595 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.596 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:16:51.616 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:51.621 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:51.628 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:16:51.630 INFO 2458 --- [ocp-schedules-10,3b2973857083b4ce,4ad34ddab3c60d27] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:16:51.632 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:51.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.634 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:51.638 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:16:51.638 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:16:51.647 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:16:51.649 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:16:51.651 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:16:51.673 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:16:51.682 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:16:51.691 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:51.692 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:16:51.693 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.693 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.693 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:16:51.693 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:16:51.694 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:51.696 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:16:51.713 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:16:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:16:51.715542+08:00, lastProcessTime=2025-10-27T11:16:41.737367+08:00 2025-10-27 11:16:51.734 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:51.734 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:51.735 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:51.736 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:51.743 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:16:51.749 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:16:51.754 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:16:51.754 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:16:51.756 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:16:51.761 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:16:51.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:51.782 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:16:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:51.791 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.791 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:16:51.794 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:16:51.796 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:16:51.797 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:16:51.806 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:16:51.807 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:51.807 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:51.816 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:16:51.832 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:51.833 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:51.834 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:51.837 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:51.837 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:51.843 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:16:51.852 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:16:51.854 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:16:51.854 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:16:51.854 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:16:51.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:51.869 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:51.871 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:16:51.873 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:16:51.894 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:16:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:51.900 INFO 2458 --- [host-check-host-status3,59fdae59ab8b35e2,adf47760b0c028ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:16:51.903 INFO 2458 --- [host-check-host-status2,59fdae59ab8b35e2,308c1eb8a49ef189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:16:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.904 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:16:51.904 INFO 2458 --- [host-check-host-status5,59fdae59ab8b35e2,dcb414b21bc0d4a9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:16:51.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:16:51.907 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:51.908 INFO 2458 --- [host-check-host-status4,59fdae59ab8b35e2,89b08c9b1d8238db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:16:51.908 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:16:51.911 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:16:51.912 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:51.917 INFO 2458 --- [host-check-host-status1,59fdae59ab8b35e2,96bb9b8b3ba10a54] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:16:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:51.977 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:51.995 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:16:52.008 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:16:51.715542+08:00 2025-10-27 11:16:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:52.039 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:16:52.039 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.040 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:16:52.042 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.043 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:16:52.064 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:16:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:52.088 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:52.089 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.089 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:16:52.107 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:52.107 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:16:52.113 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:16:52.113 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:16:52.139 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:52.139 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:16:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:16:52.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:52.214 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.214 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:16:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:16:52.229 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:16:52.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:16:52.241 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:16:52.290 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:16:52.353 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:16:52.354 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.354 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:16:52.356 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.357 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:16:52.392 INFO 2458 --- [host-check-clockdiff4,2a07bdf8663b177c,95093ce6550e61a5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535012 -1 -1 , error= 2025-10-27 11:16:52.400 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:16:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:16:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:16:52.445 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:16:52.494 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:16:52.494 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:16:52.495 INFO 2458 --- [ocp-schedules-18,fbfe83a8f180f86e,5cc7764f532bccdb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:16:52.543 INFO 2458 --- [ocp-async-7,a801d7c1d2a03d56,fc0cf1594bcae98d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:52.543 WARN 2458 --- [ocp-async-10,3c351f335d604049,bc9a99cb649c1ea7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:16:53.707 INFO 2458 --- [prometheus-minute-schedule-1,5834b4ea002f7f2c,36798eeec7f2843b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:53.717 INFO 2458 --- [prometheus-minute-schedule-1,5834b4ea002f7f2c,36798eeec7f2843b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:54.014 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_PERF_DIAGNOSER, serverId=10.40.65.146:8081, expireInSeconds=30, createTime=2025-10-27T09:38:53+08:00, updateTime=2025-10-27T11:16:23+08:00, expireTime=null, isLeader=false} 2025-10-27 11:16:54.018 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_PERF_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-10-27 11:16:54.192 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:54.193 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:54.247 INFO 2458 --- [ocp-schedules-11,00f03b71b96ee697,2bf07645b24f2719] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:16:54.278 WARN 2458 --- [scanner-112161085-1,00f03b71b96ee697,624a66a2bb6d8f22] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:16:54.323 INFO 2458 --- [ocp-schedules-27,d3a74ebcef151074,3302baf3927668df] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:16:54.328 INFO 2458 --- [ocp-schedules-27,d3a74ebcef151074,3302baf3927668df] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:16:54.543 WARN 2458 --- [ocp-async-8,97f32da92c657672,a1b0d93182f03b16] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:54.544 INFO 2458 --- [ocp-async-8,6ccb204de1b3754e,016d71cc7011ae88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:54.839 INFO 2458 --- [ocp-schedules-29,df47d5b6dd464daa,d98e414fb0a306dc] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:16:56.288 INFO 2458 --- [ocp-schedules-13,58cfcb1f64a7a281,d761db2b786014dd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:16:56.544 WARN 2458 --- [ocp-async-6,91768d2ec1303c59,02abaadb0e08e482] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:56.544 INFO 2458 --- [ocp-async-4,c5d10b051568232c,44a3368420d085ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:56.707 INFO 2458 --- [prometheus-minute-schedule-1,589a27895fa3dd1a,45909d445c213c16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:56.717 INFO 2458 --- [prometheus-minute-schedule-1,589a27895fa3dd1a,45909d445c213c16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:16:57.193 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:16:57.194 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:16:58.518 INFO 2458 --- [ocp-async-4,40b12e293c1fd7dc,a55e9774a91bfeb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:16:58.544 WARN 2458 --- [ocp-async-7,6978b8a5963a86db,d22a84400d2b5392] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:16:58.544 INFO 2458 --- [ocp-async-5,1a1ab96bbcdb4346,3d8e87b121ec9efe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:16:59.301 INFO 2458 --- [ocp-schedules-11,194f58f6cc219afe,b3ca5d68a05e5d5d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:16:59.707 INFO 2458 --- [prometheus-minute-schedule-1,dc010f5a95cb0395,de23af64344f94fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535020, interval=60 2025-10-27 11:16:59.715 INFO 2458 --- [prometheus-minute-schedule-1,dc010f5a95cb0395,de23af64344f94fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:00.194 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:00.195 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:00.544 INFO 2458 --- [ocp-async-2,43cdc42852a3a632,f3664209815dd817] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:00.544 WARN 2458 --- [ocp-async-9,85c73515ddf84c25,d9cddaaf52f2bb88] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:01.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:01.715825+08:00, lastProcessTime=2025-10-27T11:16:51.715542+08:00 2025-10-27 11:17:01.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:01.715825+08:00 2025-10-27 11:17:02.544 WARN 2458 --- [ocp-async-2,dc4a1ba74117caae,517de30258aaf602] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:02.544 INFO 2458 --- [ocp-async-2,d716443654b85f1b,68622feaed02a755] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:02.707 INFO 2458 --- [prometheus-minute-schedule-1,d22f8d13b11d1e2f,c5e726455fc9454d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:02.714 INFO 2458 --- [prometheus-minute-schedule-1,d22f8d13b11d1e2f,c5e726455fc9454d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:03.170 INFO 2458 --- [ocp-schedules-9,c11b2bd2b53d3806,2b038368634b9a6c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:17:03.177 INFO 2458 --- [ocp-schedules-9,c11b2bd2b53d3806,2b038368634b9a6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:03.195 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:03.196 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:03.221 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:03.221 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:17:03.240 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:03.241 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:17:03.255 INFO 2458 --- [ocp-schedules-9,c11b2bd2b53d3806,2b038368634b9a6c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:17:03.846 INFO 2458 --- [ocp-schedules-6,814076c94bbeadd9,528a658dbb323108] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:17:03.853 INFO 2458 --- [ocp-schedules-6,814076c94bbeadd9,528a658dbb323108] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:03.936 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:17:03.936 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:03.940 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:17:03.948 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:03.950 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:03.976 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:03.977 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:03.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:17:03.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.002 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:17:04.002 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.003 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:04.005 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.006 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.018 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.019 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:04.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.035 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.045 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.045 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:04.062 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.063 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.076 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.076 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:17:04.079 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.079 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.093 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.094 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.094 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.095 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.096 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.097 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.097 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.098 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.099 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.100 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.100 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.101 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.101 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:17:04.104 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:17:04.110 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.111 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.138 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.139 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:17:04.141 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.142 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:17:04.150 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:04.150 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.151 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:17:04.160 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.160 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:17:04.160 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.160 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:17:04.163 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:17:04.163 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:17:04.172 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.173 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:17:04.201 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.202 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:17:04.204 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.205 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:17:04.231 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.231 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:17:04.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:17:04.303 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.305 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.306 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.306 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.307 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.308 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.309 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.309 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.310 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.310 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.311 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.312 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.313 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.313 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.315 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.316 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.317 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.317 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.318 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:17:04.319 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.320 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:17:04.321 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.321 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:17:04.322 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:17:04.325 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:17:04.336 INFO 2458 --- [ocp-schedules-18,9c1c15ae10de76b9,546abaa2e4a1fe88] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:04.339 INFO 2458 --- [ocp-schedules-18,9c1c15ae10de76b9,546abaa2e4a1fe88] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:04.380 INFO 2458 --- [ocp-schedules-2,cc93cc0fbbf572e2,1422a16ab2e23a09] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:04.403 WARN 2458 --- [scanner-112161247-1,cc93cc0fbbf572e2,acbf557434f8e153] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:04.413 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:04.414 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:04.415 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:17:04.440 INFO 2458 --- [alarm-detect-4,cc93cc0fbbf572e2,71fc922d2f978611] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:04.444 INFO 2458 --- [alarm-detect-4,cc93cc0fbbf572e2,71fc922d2f978611] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:17:04.447 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:17:04.451 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:17:04.451 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:17:04.459 INFO 2458 --- [alarm-detect-3,cc93cc0fbbf572e2,d3b6f53acd277184] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:04.461 INFO 2458 --- [alarm-detect-15,cc93cc0fbbf572e2,21b28f9b4da19d31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:04.463 INFO 2458 --- [alarm-detect-3,cc93cc0fbbf572e2,d3b6f53acd277184] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:04.473 INFO 2458 --- [alarm-detect-15,cc93cc0fbbf572e2,21b28f9b4da19d31] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:04.475 INFO 2458 --- [ocp-schedules-6,814076c94bbeadd9,528a658dbb323108] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:17:04.531 INFO 2458 --- [alarm-detect-5,cc93cc0fbbf572e2,d813ae8e644569ca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:04.535 INFO 2458 --- [alarm-detect-5,cc93cc0fbbf572e2,d813ae8e644569ca] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:04.544 WARN 2458 --- [ocp-async-7,1b02790bcef4c011,cbdb0b0fc0a03d9b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:04.544 INFO 2458 --- [ocp-async-7,48f732f6f28fd538,89ad381fad40f155] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:04.581 INFO 2458 --- [alarm-detect-2,cc93cc0fbbf572e2,5468fb27b6433a9b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761534990 ~ 1761535014 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535005, value=40.399)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535005, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535005=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:17:04.588 INFO 2458 --- [alarm-detect-2,cc93cc0fbbf572e2,5468fb27b6433a9b] c.o.o.a.core.task.AlarmDetectMetricTask : alarmIds [352] will be resolved. 2025-10-27 11:17:04.588 INFO 2458 --- [alarm-detect-2,cc93cc0fbbf572e2,5468fb27b6433a9b] c.o.o.alarm.service.OcpAlarmServiceImpl : alarmIds [352] will be resolved. 2025-10-27 11:17:04.609 INFO 2458 --- [alarm-detect-2,cc93cc0fbbf572e2,5468fb27b6433a9b] c.o.o.alarm.service.OcpAlarmServiceImpl : batchUpdateStatusWithLimit done, status=Inactive, tryUpdate=1, affectRow=1, affectRowHistory=1 2025-10-27 11:17:05.707 INFO 2458 --- [prometheus-minute-schedule-1,53aa28d60565864d,c9fffc52a2e158c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:05.714 INFO 2458 --- [prometheus-minute-schedule-1,53aa28d60565864d,c9fffc52a2e158c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:06.196 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:06.198 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:06.294 INFO 2458 --- [ocp-schedules-25,b3f49c6defcbcfaf,26b17400ab3fd9a3] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:06.544 WARN 2458 --- [ocp-async-7,803c7bf0fd4aa052,fb69db04b2f7680a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:06.544 INFO 2458 --- [ocp-async-7,916899a7febe71ce,8672c8539acdf17d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:07.047 INFO 2458 --- [ocp-schedules-12,7c2ab2ceb5d9eaaa,16be64d06750c8f4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:17:07.055 INFO 2458 --- [ocp-schedules-12,7c2ab2ceb5d9eaaa,16be64d06750c8f4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:07.055 INFO 2458 --- [ocp-schedules-12,7c2ab2ceb5d9eaaa,16be64d06750c8f4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:08.518 INFO 2458 --- [ocp-async-7,29d48876c316a134,82b889e6e37017b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:08.544 WARN 2458 --- [ocp-async-6,178d0dd0c03ac933,32601bdc8fc049b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:08.544 INFO 2458 --- [ocp-async-2,d1e8105ef1f843f4,ef7c4a050635738b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:08.707 INFO 2458 --- [prometheus-minute-schedule-1,45086431143b7618,a7a9af423eb72cf6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:08.714 INFO 2458 --- [prometheus-minute-schedule-1,45086431143b7618,a7a9af423eb72cf6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:09.198 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:09.199 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:10.011 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:17:10.013 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:17:10.048 INFO 2458 --- [Check obproxy1,87f005c7ecae386e,b23b0961faed251f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:17:10.048 INFO 2458 --- [Check obproxy1,87f005c7ecae386e,b23b0961faed251f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:17:10.049 INFO 2458 --- [Collect obproxy1,fa91f5ae59ffc26e,04b3ac1c932f90ce] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:17:10.049 INFO 2458 --- [Collect obproxy1,fa91f5ae59ffc26e,04b3ac1c932f90ce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:17:10.078 INFO 2458 --- [Collect obproxy1,fa91f5ae59ffc26e,04b3ac1c932f90ce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:17:10.080 INFO 2458 --- [Check obproxy1,87f005c7ecae386e,b23b0961faed251f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:17:10.081 INFO 2458 --- [Collect obproxy1,fa91f5ae59ffc26e,04b3ac1c932f90ce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:17:10.083 INFO 2458 --- [Check obproxy1,87f005c7ecae386e,b23b0961faed251f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:17:10.083 INFO 2458 --- [Check obproxy1,87f005c7ecae386e,b23b0961faed251f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:17:10.084 INFO 2458 --- [Collect obproxy1,fa91f5ae59ffc26e,04b3ac1c932f90ce] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:17:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:17:10.545 INFO 2458 --- [ocp-async-9,ba8dd9d93f59dfe0,edbd562064f31df3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:10.545 WARN 2458 --- [ocp-async-4,527b1f237009db14,4ce81a52c4042251] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:10.932 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761534960, nodeCount=821, affectRows=821 2025-10-27 11:17:11.707 INFO 2458 --- [prometheus-minute-schedule-1,bc6d8fb5e12676ae,d7d46e8d04036983] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:11.714 INFO 2458 --- [prometheus-minute-schedule-1,bc6d8fb5e12676ae,d7d46e8d04036983] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:11.713791+08:00, lastProcessTime=2025-10-27T11:17:01.715825+08:00 2025-10-27 11:17:11.933 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:11.713791+08:00 2025-10-27 11:17:12.199 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:12.200 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:12.545 INFO 2458 --- [ocp-async-8,b253823278a1b7e5,4f7838681ef27393] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:12.545 WARN 2458 --- [ocp-async-1,f494d311641204b1,36d58e21f2285b85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:14.348 INFO 2458 --- [ocp-schedules-7,750dc78347c5a284,c94cded2efe130cf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:14.351 INFO 2458 --- [ocp-schedules-7,750dc78347c5a284,c94cded2efe130cf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:14.420 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:17:14.420 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:17:14.429 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.429 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:17:14.443 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.443 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.445 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.445 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:17:14.455 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.455 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.456 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.456 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:17:14.462 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:17:14.462 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:17:14.463 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.463 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:17:14.465 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.466 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.467 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.467 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:17:14.488 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.488 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.489 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.490 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:17:14.505 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.506 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.506 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.507 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:17:14.520 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.520 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.521 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.522 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:17:14.535 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.536 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.536 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.537 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:17:14.545 INFO 2458 --- [ocp-async-8,d685d7ee3a3431e0,c90027e8feaf0984] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:14.545 WARN 2458 --- [ocp-async-6,9fb2181af6ce197b,00f5055cc8babbc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:14.551 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.551 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.552 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.553 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:17:14.567 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.568 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.569 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.569 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:17:14.583 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.584 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.584 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.585 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:17:14.592 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.592 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.593 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.594 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:17:14.601 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.601 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.602 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.602 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:17:14.608 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.608 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.609 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.610 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:17:14.615 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.616 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.616 INFO 2458 --- [ocp-schedules-24,54d45ce55b6c115b,05c6bcf5ea941d34] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:14.617 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.618 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:17:14.627 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.627 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.628 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.629 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:17:14.639 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.641 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.642 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.642 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:17:14.657 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.657 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.659 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.661 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:17:14.664 WARN 2458 --- [scanner-112163053-1,54d45ce55b6c115b,85772661a05f8fe2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:14.668 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.669 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.670 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.671 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:17:14.678 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.679 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.680 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.681 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:17:14.688 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.689 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.689 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.690 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:17:14.705 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.706 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.707 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.708 INFO 2458 --- [prometheus-minute-schedule-1,5794d1c678283d39,d276f6869c25a110] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:14.708 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:17:14.721 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.722 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.723 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.723 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:17:14.736 INFO 2458 --- [prometheus-minute-schedule-1,5794d1c678283d39,d276f6869c25a110] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:14.742 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.742 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.745 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.746 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:17:14.753 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.754 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.754 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.755 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:17:14.763 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.763 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.764 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.765 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:17:14.772 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.773 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.774 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.774 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:17:14.781 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.781 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.782 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.784 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:17:14.791 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.792 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.793 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.793 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:17:14.801 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.801 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.804 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.804 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:17:14.811 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.812 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.812 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.813 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:17:14.815 INFO 2458 --- [alarm-detect-1,54d45ce55b6c115b,385f292bb2ee52f2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535005 ~ 1761535024 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535020, value=43.785)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535020, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535020=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:17:14.819 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.820 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.821 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.821 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:17:14.827 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.827 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.828 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.829 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:17:14.834 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.835 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.835 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.836 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:17:14.842 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.843 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.843 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.844 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:17:14.849 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.849 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.850 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.851 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:17:14.856 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.856 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.857 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.858 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:17:14.864 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.864 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.865 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.865 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:17:14.871 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.871 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.872 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.873 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:17:14.878 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.878 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.879 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.880 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:17:14.885 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.885 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.886 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.886 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:17:14.892 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.892 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.893 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.894 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:17:14.899 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.899 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.900 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.901 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:17:14.906 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.907 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.907 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.908 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:17:14.913 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.914 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.915 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.915 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:17:14.924 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.924 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.925 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.925 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:17:14.933 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.933 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.934 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.935 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:17:14.940 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.941 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.942 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.942 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:17:14.947 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.948 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.948 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.949 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:17:14.955 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.955 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.956 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.957 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:17:14.962 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.962 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.963 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.963 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:17:14.969 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.969 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.970 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.970 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:17:14.971 INFO 2458 --- [ocp-schedules-28,f65fb97d4a4b94c7,591a208ec866393c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:17:14.971+08:00, clusters=[1, 2] 2025-10-27 11:17:14.973 INFO 2458 --- [ocp-schedules-28,f65fb97d4a4b94c7,591a208ec866393c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:17:14.974 INFO 2458 --- [ocp-schedules-28,f65fb97d4a4b94c7,591a208ec866393c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:17:14.975 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.976 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.977 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.977 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:17:14.982 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.983 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.984 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.984 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:17:14.989 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.990 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.991 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.991 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:17:14.996 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.996 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:14.997 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:14.998 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:17:15.003 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.003 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:15.004 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.005 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:17:15.009 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.010 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:15.011 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.011 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:17:15.015 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.016 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:15.017 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.017 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:17:15.022 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.023 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:17:15.023 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.024 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:17:15.028 INFO 2458 --- [ocp-schedules-22,4863826af057ecff,e48268644d888dbd] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:17:15.147 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=1000, affectRows=1000 2025-10-27 11:17:15.154 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=994, validCount=991 2025-10-27 11:17:15.200 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:15.201 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:15.249 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=1000, affectRows=1000 2025-10-27 11:17:15.253 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=999, validCount=996 2025-10-27 11:17:15.369 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=1000, affectRows=1000 2025-10-27 11:17:15.374 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=999, validCount=999 2025-10-27 11:17:15.394 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505650) Duplicate entry '835370602979598336-1761534900' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505650) Duplicate entry '835370602979598336-1761534900' for key 'PRIMARY' 2025-10-27 11:17:15.714 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=99, affectRows=99 2025-10-27 11:17:15.714 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=98, validCount=98 2025-10-27 11:17:15.717 INFO 2458 --- [ocp-schedules-13,152b7b8eb0cca1d9,0ef5c92643cf1994] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:17:15.717 INFO 2458 --- [ocp-schedules-13,152b7b8eb0cca1d9,0ef5c92643cf1994] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:17:15.720 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=1000, affectRows=1000 2025-10-27 11:17:15.724 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=1000, validCount=993 2025-10-27 11:17:15.774 INFO 2458 --- [Collect all tenants major compactions2,152b7b8eb0cca1d9,5014c019f089bbef] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:17:15.775 INFO 2458 --- [Collect all tenants major compactions3,152b7b8eb0cca1d9,18e680364fd841d6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:17:15.775 INFO 2458 --- [Collect all tenants major compactions1,152b7b8eb0cca1d9,85990bd66d6e79f2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:17:15.775 INFO 2458 --- [Collect all tenants major compactions5,152b7b8eb0cca1d9,cba98e68b9968af9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:17:15.775 INFO 2458 --- [Collect all tenants major compactions4,152b7b8eb0cca1d9,b018f0074a8734e4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:17:15.778 INFO 2458 --- [Collect all tenants major compactions6,152b7b8eb0cca1d9,24ced29d96dd6a04] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:17:15.814 INFO 2458 --- [Collect all tenants major compactions5,152b7b8eb0cca1d9,cba98e68b9968af9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.814 INFO 2458 --- [Collect all tenants major compactions6,152b7b8eb0cca1d9,24ced29d96dd6a04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.815 INFO 2458 --- [Collect all tenants major compactions6,152b7b8eb0cca1d9,24ced29d96dd6a04] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:17:15.815 INFO 2458 --- [Collect all tenants major compactions5,152b7b8eb0cca1d9,cba98e68b9968af9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:17:15.819 INFO 2458 --- [Collect all tenants major compactions6,152b7b8eb0cca1d9,24ced29d96dd6a04] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:17:15.819 INFO 2458 --- [Collect all tenants major compactions5,152b7b8eb0cca1d9,cba98e68b9968af9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:17:15.829 INFO 2458 --- [Collect all tenants major compactions4,152b7b8eb0cca1d9,b018f0074a8734e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.830 INFO 2458 --- [Collect all tenants major compactions4,152b7b8eb0cca1d9,b018f0074a8734e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:17:15.834 INFO 2458 --- [Collect all tenants major compactions4,152b7b8eb0cca1d9,b018f0074a8734e4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:17:15.834 INFO 2458 --- [Collect all tenants major compactions2,152b7b8eb0cca1d9,5014c019f089bbef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.835 INFO 2458 --- [Collect all tenants major compactions2,152b7b8eb0cca1d9,5014c019f089bbef] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:17:15.839 INFO 2458 --- [Collect all tenants major compactions2,152b7b8eb0cca1d9,5014c019f089bbef] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:17:15.839 INFO 2458 --- [Collect all tenants major compactions3,152b7b8eb0cca1d9,18e680364fd841d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.840 INFO 2458 --- [Collect all tenants major compactions1,152b7b8eb0cca1d9,85990bd66d6e79f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:15.840 INFO 2458 --- [Collect all tenants major compactions3,152b7b8eb0cca1d9,18e680364fd841d6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:17:15.842 INFO 2458 --- [Collect all tenants major compactions1,152b7b8eb0cca1d9,85990bd66d6e79f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:17:15.843 INFO 2458 --- [Collect all tenants major compactions3,152b7b8eb0cca1d9,18e680364fd841d6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:17:15.846 INFO 2458 --- [Collect all tenants major compactions1,152b7b8eb0cca1d9,85990bd66d6e79f2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:17:15.847 INFO 2458 --- [ocp-schedules-13,152b7b8eb0cca1d9,0ef5c92643cf1994] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:17:16.301 INFO 2458 --- [ocp-schedules-12,7b944a726a472eb8,ebc7006e40e01123] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:16.545 INFO 2458 --- [ocp-async-4,b0c485b5e9509d1a,230022bbe519b086] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:16.545 WARN 2458 --- [ocp-async-5,66a60e241c23969c,5687732ddaddd8ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:17.707 INFO 2458 --- [prometheus-minute-schedule-1,2b903b36793fcea7,8d576a9f36a06b6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:17.714 INFO 2458 --- [prometheus-minute-schedule-1,2b903b36793fcea7,8d576a9f36a06b6d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:18.201 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:18.201 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:18.215 INFO 2458 --- [ocp-schedules-10,f6472c8a682b62a4,7374e8d4d62e6e0a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:17:18.215 INFO 2458 --- [ocp-schedules-10,f6472c8a682b62a4,7374e8d4d62e6e0a] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:17:18.223 INFO 2458 --- [ocp-schedules-10,f6472c8a682b62a4,7374e8d4d62e6e0a] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:17:18.223 INFO 2458 --- [ocp-schedules-10,f6472c8a682b62a4,7374e8d4d62e6e0a] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:17:18.518 INFO 2458 --- [ocp-async-4,e6d24880e653764e,c345226a4206d9ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:18.545 INFO 2458 --- [ocp-async-1,bd56835c780851e1,a246c41b2597fe94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:18.545 WARN 2458 --- [ocp-async-7,56adf2a03ad94a19,8ad9ca027e6bdb07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:18.989 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,59713d958f9c772f,3359333dfec9bddb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=59713d958f9c772f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:17:18.989 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,59713d958f9c772f,3359333dfec9bddb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=59713d958f9c772f, duration=1 ms] 2025-10-27 11:17:19.752 INFO 2458 --- [ocp-schedules-2,4a616fdea3692158,d9912673257d13c5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:17:20.017 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761534960, nodeCount=1000, affectRows=1000 2025-10-27 11:17:20.022 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761534960, nodeCount=992, validCount=978 2025-10-27 11:17:20.545 INFO 2458 --- [ocp-async-6,bbc92aaef7e24754,29ef94cde172a80b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:20.545 WARN 2458 --- [ocp-async-8,ef08b9d990a8dfe0,4bf4e51f1aa6e046] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:20.707 INFO 2458 --- [prometheus-minute-schedule-1,1eeec40668f57613,f3d40341784358d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:20.713 INFO 2458 --- [prometheus-minute-schedule-1,1eeec40668f57613,f3d40341784358d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:21.201 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:21.202 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:21.408 INFO 2458 --- [ocp-schedules-12,f01e598a51e49dc3,126d1d5a6b5df7f3] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:21.408 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:17:21.408 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:17:21.408 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:17:21.408 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:17:21.409 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:17:21.410 INFO 2458 --- [ocp-schedules-7,589f68b4226c9fdd,7c123b15fd3297d0] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:21.410 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:17:21.411 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:17:21.411 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:17:21.411 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:17:21.411 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:17:21.424 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:21.424 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:21.424 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:21.424 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:21.424 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:21.425 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:21.425 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:21.426 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:21.429 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:21.429 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:21.430 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:21.430 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:21.430 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:21.430 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:21.430 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:21.431 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:21.431 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:21.433 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:21.433 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:21.435 INFO 2458 --- [ocp-schedules-11,13d74f2a09448750,6afa001909546807] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:21.435 INFO 2458 --- [host-check-clockdiff1,13d74f2a09448750,3d44c33b8304e9db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:17:21.435 INFO 2458 --- [host-check-clockdiff2,13d74f2a09448750,7f3ea02ca4c9cfa8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:17:21.436 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:21.436 INFO 2458 --- [host-check-clockdiff3,13d74f2a09448750,3ec9e884f7282646] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:17:21.436 INFO 2458 --- [host-check-clockdiff4,13d74f2a09448750,62931ee50533cf58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:17:21.437 INFO 2458 --- [host-check-clockdiff5,13d74f2a09448750,51d6c79883bf0944] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:17:21.455 INFO 2458 --- [ocp-schedules-26,a2bb493d31c6343b,78fefc131ea21e45] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:17:21.470 INFO 2458 --- [host-check-clockdiff1,13d74f2a09448750,3d44c33b8304e9db] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535041 0 0 , error= 2025-10-27 11:17:21.471 INFO 2458 --- [host-check-clockdiff5,13d74f2a09448750,51d6c79883bf0944] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535041 0 0 , error= 2025-10-27 11:17:21.473 INFO 2458 --- [host-check-clockdiff3,13d74f2a09448750,3ec9e884f7282646] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535041 0 0 , error= 2025-10-27 11:17:21.476 INFO 2458 --- [host-check-clockdiff4,13d74f2a09448750,62931ee50533cf58] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535041 0 0 , error= 2025-10-27 11:17:21.509 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:17:21.510 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:17:21.510 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:17:21.512 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:17:21.511884934+08:00 2025-10-27 11:17:21.512 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:17:21.512+08:00 2025-10-27 11:17:21.512 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:17:21.513 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:17:21.513 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:17:21.513 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:17:21.513 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:17:21.513 INFO 2458 --- [host-check-agent-status1,f01e598a51e49dc3,3cab61b560d4ef87] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:17:21.514 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:17:21.514 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:17:21.514 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:17:21.514 INFO 2458 --- [host-check-agent-status4,f01e598a51e49dc3,5eeb4291fb7b69c7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:17:21.515 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:17:21.515813405+08:00 2025-10-27 11:17:21.516 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:17:21.515+08:00 2025-10-27 11:17:21.516 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status3,f01e598a51e49dc3,d9e64cca8a6730ff] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status5,f01e598a51e49dc3,7752d44afab661ec] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:17:21.517 INFO 2458 --- [host-check-agent-status2,f01e598a51e49dc3,3e2572657d8f9103] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:17:21.518 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:17:21.519 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:17:21.519 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:17:21.518480589+08:00 2025-10-27 11:17:21.519 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:17:21.518265713+08:00 2025-10-27 11:17:21.519 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:17:21.519+08:00 2025-10-27 11:17:21.520 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:17:21.519+08:00 2025-10-27 11:17:21.524 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:17:21.512+08:00 and status AVAILABLE 2025-10-27 11:17:21.525 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:17:21.526 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:17:21.525473126+08:00 2025-10-27 11:17:21.526 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:17:21.526+08:00 2025-10-27 11:17:21.527 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:17:21.519+08:00 and status AVAILABLE 2025-10-27 11:17:21.527 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:17:21.515+08:00 and status AVAILABLE 2025-10-27 11:17:21.528 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:17:21.519+08:00 and status AVAILABLE 2025-10-27 11:17:21.528 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:17:21.531 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:17:21.533 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:17:21.533 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:17:21.533 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:17:21.526+08:00 and status AVAILABLE 2025-10-27 11:17:21.538 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:17:21.545 INFO 2458 --- [ocp-schedules-29,9f09fa92b7d5e26a,81b03d9a36b4745c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:17:21.548 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:21.549 INFO 2458 --- [ocp-schedules-21,c4c55e8016b759ad,a5696eb0df666736] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:17:21.550 INFO 2458 --- [ocp-schedules-29,9f09fa92b7d5e26a,81b03d9a36b4745c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:21.551 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:21.551 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:21.553 INFO 2458 --- [ocp-schedules-21,c4c55e8016b759ad,a5696eb0df666736] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:21.554 INFO 2458 --- [ocp-schedules-22,389887e45275945a,4490b87b7d621a99] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:17:21.554 INFO 2458 --- [ocp-schedules-21,c4c55e8016b759ad,a5696eb0df666736] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:17:21.554 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:21.555 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:21.556 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:21.557 INFO 2458 --- [ocp-schedules-22,389887e45275945a,4490b87b7d621a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:21.558 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:21.558 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:21.558 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:21.561 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:21.567 INFO 2458 --- [ocp-schedules-27,2df3653be0f02543,510b6bf9224a9f31] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:17:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:21.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:17:21.612 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.613 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:21.613 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:21.626 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:17:21.626 INFO 2458 --- [ocp-schedules-29,9f09fa92b7d5e26a,81b03d9a36b4745c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:17:21.629 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:21.630 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:21.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:21.637 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:21.638 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:21.646 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:17:21.647 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:17:21.650 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:17:21.657 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:17:21.657 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:17:21.684 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:17:21.689 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:21.689 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.689 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:17:21.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:17:21.691 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:17:21.693 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:17:21.693 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.694 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:21.697 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:17:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:21.714368+08:00, lastProcessTime=2025-10-27T11:17:11.713791+08:00 2025-10-27 11:17:21.722 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:21.728 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:21.728 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:21.735 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:21.743 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:17:21.744 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:21.746 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:17:21.751 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:17:21.758 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:17:21.764 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:17:21.774 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:21.775 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.775 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:17:21.782 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:17:21.784 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:21.785 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.785 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.787 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:17:21.789 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:17:21.790 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:21.793 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:17:21.793 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:17:21.815 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:17:21.818 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:21.820 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:21.820 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:21.824 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:21.826 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:21.831 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:21.840 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:17:21.845 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:17:21.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:21.850 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:17:21.856 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:17:21.861 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:21.878 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:17:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:21.886 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:21.887 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:21.887 INFO 2458 --- [host-check-host-status4,589f68b4226c9fdd,2322363ba6de7c87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:17:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:21.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:21.889 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:17:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:17:21.892 INFO 2458 --- [host-check-host-status2,589f68b4226c9fdd,c5d245564baae96b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:17:21.892 INFO 2458 --- [host-check-host-status1,589f68b4226c9fdd,18b829311e929d00] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:17:21.893 INFO 2458 --- [host-check-host-status3,589f68b4226c9fdd,bcb18443238ccf05] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:17:21.894 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:21.899 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:17:21.909 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:21.911 INFO 2458 --- [host-check-host-status5,589f68b4226c9fdd,35d01f270358c903] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:17:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:21.936 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:21.999 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:17:22.014 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:22.015 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:22.032 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:21.714368+08:00 2025-10-27 11:17:22.044 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:17:22.045 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.046 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:17:22.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:17:22.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:22.051 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:22.064 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:17:22.089 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:22.089 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.090 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:17:22.090 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:22.090 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:22.112 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:17:22.112 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:17:22.122 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:22.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:22.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:22.198 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:22.198 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:22.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:17:22.212 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:17:22.221 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:22.224 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:22.270 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:17:22.336 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:17:22.336 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.337 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:17:22.339 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.340 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:17:22.382 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:17:22.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:22.439 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:22.439 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:17:22.492 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:17:22.492 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:17:22.492 INFO 2458 --- [ocp-schedules-22,389887e45275945a,4490b87b7d621a99] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:17:22.545 INFO 2458 --- [ocp-async-10,86efa46542cd4723,973ff151209b56a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:22.545 WARN 2458 --- [ocp-async-2,569ecc8041cca188,6db2a1163cc22ce0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:23.233 INFO 2458 --- [host-check-clockdiff2,13d74f2a09448750,7f3ea02ca4c9cfa8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535043 -1 -1 , error= 2025-10-27 11:17:23.707 INFO 2458 --- [prometheus-minute-schedule-1,c80c01dbfdd271c0,d164b17386ef4640] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:23.715 INFO 2458 --- [prometheus-minute-schedule-1,c80c01dbfdd271c0,d164b17386ef4640] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:24.009 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=PLAN_CHANGE_DETECTOR, serverId=10.40.65.146:8081, expireInSeconds=30, createTime=2025-10-27T09:39:23+08:00, updateTime=2025-10-27T11:16:53+08:00, expireTime=null, isLeader=false} 2025-10-27 11:17:24.011 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service PLAN_CHANGE_DETECTOR, Incorrect result size: expected 1, actual 0 2025-10-27 11:17:24.202 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:24.203 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:24.359 INFO 2458 --- [ocp-schedules-28,ced5779f846e03a7,b7432a0a7ca236c4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:24.362 INFO 2458 --- [ocp-schedules-28,ced5779f846e03a7,b7432a0a7ca236c4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:24.545 INFO 2458 --- [ocp-async-7,8d3c70ce81b74e6f,d87cd72a5d82e355] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:24.546 WARN 2458 --- [ocp-async-1,0803778ecc8ca13f,8267ba42bf3355eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:24.825 INFO 2458 --- [ocp-schedules-23,c88997d99a3c4797,42fae372fafc6a6c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:24.894 INFO 2458 --- [alarm-detect-5,c88997d99a3c4797,aeac0a447b08fb0c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:24.898 INFO 2458 --- [alarm-detect-5,c88997d99a3c4797,aeac0a447b08fb0c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:24.904 INFO 2458 --- [alarm-detect-2,c88997d99a3c4797,cf70dd5f42109ffd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:24.909 INFO 2458 --- [alarm-detect-12,c88997d99a3c4797,b750b930069f14aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:24.910 INFO 2458 --- [alarm-detect-2,c88997d99a3c4797,cf70dd5f42109ffd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:24.912 WARN 2458 --- [scanner-112163273-1,c88997d99a3c4797,f92c9f9320059b79] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:24.913 INFO 2458 --- [alarm-detect-12,c88997d99a3c4797,b750b930069f14aa] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:24.990 INFO 2458 --- [alarm-detect-12,c88997d99a3c4797,079c5d28ede94d14] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:24.995 INFO 2458 --- [alarm-detect-12,c88997d99a3c4797,079c5d28ede94d14] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:26.319 INFO 2458 --- [ocp-schedules-20,c5dab1734287c81a,d10d7f08bd76b4dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:26.546 INFO 2458 --- [ocp-async-7,9a52a7879da617af,4efd38eb14819ec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:26.546 WARN 2458 --- [ocp-async-3,0b864f62ad337fbf,c79d1d4cd2e4ae62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:26.707 INFO 2458 --- [prometheus-minute-schedule-1,1ec0cec192404b1b,017aea7237d76496] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:26.714 INFO 2458 --- [prometheus-minute-schedule-1,1ec0cec192404b1b,017aea7237d76496] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:27.057 INFO 2458 --- [ocp-schedules-25,b3744e7b55254596,e00ffca65b25db6f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:17:27.065 INFO 2458 --- [ocp-schedules-25,b3744e7b55254596,e00ffca65b25db6f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:27.065 INFO 2458 --- [ocp-schedules-25,b3744e7b55254596,e00ffca65b25db6f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:27.203 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:27.204 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:28.518 INFO 2458 --- [ocp-async-2,3aa404b4c058d0c8,e1819f7286019816] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:28.546 INFO 2458 --- [ocp-async-5,a9dcee43650e7ef6,d499320072ac6d20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:28.546 WARN 2458 --- [ocp-async-10,738787895f8e80fe,f443d04e709682ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:28.924 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:17:28.924 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:17:28.925 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:17:28.925 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:17:28.927 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:17:28.927 INFO 2458 --- [ocp-schedules-22,925f615a47fb6294,2f04375302fdfa4e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:17:29.386 INFO 2458 --- [ocp-schedules-5,8539bb465714d2f9,43aed846a5ef76f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:17:29.391 INFO 2458 --- [ocp-schedules-5,8539bb465714d2f9,43aed846a5ef76f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:17:29.391 INFO 2458 --- [ocp-schedules-5,8539bb465714d2f9,43aed846a5ef76f4] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:17:29.707 INFO 2458 --- [prometheus-minute-schedule-1,034f5be17dff431b,f1e7aec69689c4fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:29.716 INFO 2458 --- [prometheus-minute-schedule-1,034f5be17dff431b,f1e7aec69689c4fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:30.204 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:30.205 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:30.546 INFO 2458 --- [ocp-async-9,2cc6b43379a398cd,fd4f430743dd13fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:30.546 WARN 2458 --- [ocp-async-4,744de45040005b61,b8d6753c8197b3dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:31.180 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:17:31.180 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:17:31.185 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:17:31.185 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:17:31.185 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:17:31.185 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:17:31.189 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:17:31.189 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:17:31.189 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:17:31.189 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:17:31.230 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:17:31.237 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:17:31.237 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:17:31.237 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:17:31.237 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:17:31.237 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:17:31.245 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:17:31.647 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:17:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:31.715428+08:00, lastProcessTime=2025-10-27T11:17:21.714368+08:00 2025-10-27 11:17:31.944 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:31.715428+08:00 2025-10-27 11:17:32.051 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.862327112S 2025-10-27 11:17:32.051 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:17:32.123 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:17:32.123 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:17:32.154 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:17:32.196 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:17:32.205 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:17:32.342 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] o.s.boot.SpringApplication : Started application in 1.157 seconds (JVM running for 257520.769) 2025-10-27 11:17:32.342 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.157720449S 2025-10-27 11:17:32.546 INFO 2458 --- [ocp-async-3,6cf80729710b9002,fede24093cd4a27a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:32.546 WARN 2458 --- [ocp-async-8,dc3f45170f975bc5,2ac5da0cb923d162] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:32.683 INFO 2458 --- [ocp-schedules-30,ec8cbf0482a7d847,d4f3d55f8f9d27f9] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:17:32.707 INFO 2458 --- [prometheus-minute-schedule-1,55155a0c9fdc67ce,343215eb890c67d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:32.715 INFO 2458 --- [prometheus-minute-schedule-1,55155a0c9fdc67ce,343215eb890c67d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:33.205 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:33.206 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:33.824 INFO 2458 --- [ocp-schedules-11,f2690b4db08a1b7e,8a1306d992cfb025] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:17:33.824 INFO 2458 --- [ocp-schedules-11,f2690b4db08a1b7e,8a1306d992cfb025] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:17:34.370 INFO 2458 --- [ocp-schedules-16,0276509ad9a9ede4,683d0635ddc9c68a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:34.373 INFO 2458 --- [ocp-schedules-16,0276509ad9a9ede4,683d0635ddc9c68a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:34.546 INFO 2458 --- [ocp-async-2,0366c8f8fbe7aa5f,3d20bd4643611769] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:34.546 WARN 2458 --- [ocp-async-10,e0640c5962911a65,d8fce23d6460764c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:35.023 INFO 2458 --- [ocp-schedules-1,96e89d53aca26704,c259b9cefc2bb259] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:35.050 WARN 2458 --- [scanner-112163493-1,96e89d53aca26704,2cf69c794ba72614] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:35.129 INFO 2458 --- [alarm-detect-19,96e89d53aca26704,1829cc7e1e31615e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535020 ~ 1761535045 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535035, value=52.341)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535035, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535035=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:17:35.707 INFO 2458 --- [prometheus-minute-schedule-1,54926ca58f384976,d8d66a1a48b4a0e5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:35.715 INFO 2458 --- [prometheus-minute-schedule-1,54926ca58f384976,d8d66a1a48b4a0e5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:35.942 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,f319414fbda923a9,9aa0807c33a2bc85] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=f319414fbda923a9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:17:35.942 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,f319414fbda923a9,9aa0807c33a2bc85] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=f319414fbda923a9, duration=2 ms] 2025-10-27 11:17:36.206 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:36.207 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:36.326 INFO 2458 --- [ocp-schedules-4,52b4c1744d56bbd8,23edc1b2080deea7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:36.546 INFO 2458 --- [ocp-async-5,98dd75bf2023b4c6,e55a950d7dce046f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:36.546 WARN 2458 --- [ocp-async-4,b8148f38ed93c541,0d15289f98722988] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:36.987 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:17:36.987 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:17:37.098 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:37.099 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:17:37.114 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:37.114 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:17:37.176 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:37.177 INFO 2458 --- [ocp-schedules-10,f9d64eddd1927cc6,9deef53c19637b1f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:17:37.708 INFO 2458 --- [prometheus-cache-monitor1,158f328dfabaab07,868736f5604f04b6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20379,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:17:37.708 INFO 2458 --- [prometheus-cache-monitor1,158f328dfabaab07,868736f5604f04b6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:17:38.197 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:38.197 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:17:38.197 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:17:38.197 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:17:38.198 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:17:38.198 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:17:38.209 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:38.209 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:38.211 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:38.211 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:38.211 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:38.214 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:38.215 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:38.216 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:38.216 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:38.216 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:38.275 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:17:38.277 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:17:38.279 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:17:38.281 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:17:38.282 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:17:38.291 INFO 2458 --- [ocp-schedules-28,5cfb4b321ba1c127,2de61045ec58dd58] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:17:38.519 INFO 2458 --- [ocp-async-9,6a4b0479770f0e36,cb5d2719f18a0040] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:38.546 WARN 2458 --- [ocp-async-6,6aec89797c82d5a6,231dad518a9b9d6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:38.546 INFO 2458 --- [ocp-async-8,a9214668f5739423,4903e14d675de474] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:38.670 INFO 2458 --- [ocp-schedules-5,136b0e51a0ffe45f,e295c1bef084c700] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:17:38.676 INFO 2458 --- [ocp-schedules-5,136b0e51a0ffe45f,e295c1bef084c700] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:38.676 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:17:38.676 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:17:38.677 INFO 2458 --- [ocp-schedules-5,136b0e51a0ffe45f,e295c1bef084c700] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:17:38.707 INFO 2458 --- [prometheus-minute-schedule-1,d14dd8348e521777,06ccbc603014c2ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:38.715 INFO 2458 --- [prometheus-minute-schedule-1,d14dd8348e521777,06ccbc603014c2ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:39.207 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:39.208 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:40.015 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:17:40.016 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:17:40.049 INFO 2458 --- [Check obproxy1,946b628a679ae9c1,855e01ad21a5e654] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:17:40.053 INFO 2458 --- [Check obproxy1,946b628a679ae9c1,855e01ad21a5e654] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:17:40.055 INFO 2458 --- [Collect obproxy1,16c1732c59efec6c,d47872958f09efae] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:17:40.055 INFO 2458 --- [Collect obproxy1,16c1732c59efec6c,d47872958f09efae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:17:40.083 INFO 2458 --- [Collect obproxy1,16c1732c59efec6c,d47872958f09efae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:17:40.083 INFO 2458 --- [Check obproxy1,946b628a679ae9c1,855e01ad21a5e654] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:17:40.086 INFO 2458 --- [Collect obproxy1,16c1732c59efec6c,d47872958f09efae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:17:40.087 INFO 2458 --- [Check obproxy1,946b628a679ae9c1,855e01ad21a5e654] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:17:40.087 INFO 2458 --- [Check obproxy1,946b628a679ae9c1,855e01ad21a5e654] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:17:40.089 INFO 2458 --- [Collect obproxy1,16c1732c59efec6c,d47872958f09efae] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:17:40.547 WARN 2458 --- [ocp-async-10,2ef8b475e8b76a0f,8786a24d9be57a6e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:40.547 INFO 2458 --- [ocp-async-7,ee0f885109b3c060,c6b66ca01bc3cbc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:40.973 INFO 2458 --- [ocp-schedules-14,7d1353d80ca6d307,ea1c6bf4bce1e771] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:17:40.974 INFO 2458 --- [ocp-schedules-14,7d1353d80ca6d307,ea1c6bf4bce1e771] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:17:41.707 INFO 2458 --- [prometheus-minute-schedule-1,4c2fdb7117729f50,d20d84aa23aea1f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:41.714 INFO 2458 --- [prometheus-minute-schedule-1,4c2fdb7117729f50,d20d84aa23aea1f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:41.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:41.716046+08:00, lastProcessTime=2025-10-27T11:17:31.715428+08:00 2025-10-27 11:17:41.935 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:41.716046+08:00 2025-10-27 11:17:42.208 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:42.210 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:42.547 WARN 2458 --- [ocp-async-1,e2a9fb69b38fcf9c,6630e88f8b50d2c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:42.547 INFO 2458 --- [ocp-async-4,c395ce409d0f3807,e2a6cbf3f05f8953] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:44.088 INFO 2458 --- [ocp-schedules-3,c767fa17e4bdb7fc,821d153dc8c9ec1b] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:17:44.384 INFO 2458 --- [ocp-schedules-21,5cdf49d47ff21683,5e7b5aff705c7f3a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:44.388 INFO 2458 --- [ocp-schedules-21,5cdf49d47ff21683,5e7b5aff705c7f3a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:44.547 WARN 2458 --- [ocp-async-6,f653fd04681a051f,e8343b47b8df7df5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:44.547 INFO 2458 --- [ocp-async-9,bbf4d47210611d28,ff5673874c26c280] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:44.707 INFO 2458 --- [prometheus-minute-schedule-1,e53e61166ab67923,0236d91b365dbd0e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:44.714 INFO 2458 --- [prometheus-minute-schedule-1,e53e61166ab67923,0236d91b365dbd0e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:45.102 INFO 2458 --- [prometheus-second-schedule-1,197e38c1e52c6e14,9f8755a2b62a6b5e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:17:45.143 INFO 2458 --- [ocp-schedules-8,41befb36a819b0f8,ce5dd07e0746f1c9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:45.193 WARN 2458 --- [scanner-112163669-1,41befb36a819b0f8,e732826173cd76ba] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:45.210 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:45.211 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:45.212 INFO 2458 --- [alarm-detect-6,41befb36a819b0f8,f5dd98cd09759628] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:45.217 INFO 2458 --- [alarm-detect-6,41befb36a819b0f8,f5dd98cd09759628] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:17:45.224 INFO 2458 --- [alarm-detect-13,41befb36a819b0f8,004e50bb1c61a285] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:45.228 INFO 2458 --- [alarm-detect-13,41befb36a819b0f8,004e50bb1c61a285] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:17:45.230 INFO 2458 --- [alarm-detect-2,41befb36a819b0f8,b78b55e31a42bcbd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:45.234 INFO 2458 --- [alarm-detect-2,41befb36a819b0f8,b78b55e31a42bcbd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:17:45.247 INFO 2458 --- [alarm-detect-8,41befb36a819b0f8,099e004fa2932fbe] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761534995 ~ 1761535055 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535020, value=203354.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203354.56, values={1761535020=203354.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:17:45.248 INFO 2458 --- [alarm-detect-8,41befb36a819b0f8,099e004fa2932fbe] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:17:45.288 INFO 2458 --- [alarm-detect-11,41befb36a819b0f8,af4a3ec9d20d3273] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761534995 ~ 1761535055 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535020, value=93.063)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.063, values={1761535020=93.063}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:17:45.289 INFO 2458 --- [alarm-detect-11,41befb36a819b0f8,af4a3ec9d20d3273] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:17:45.297 INFO 2458 --- [alarm-detect-12,41befb36a819b0f8,18ed4700ae621fe9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761534995 ~ 1761535055 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535020, value=203355.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203355.912, values={1761535020=203355.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:17:45.297 INFO 2458 --- [alarm-detect-12,41befb36a819b0f8,18ed4700ae621fe9] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:17:45.298 INFO 2458 --- [alarm-detect-4,41befb36a819b0f8,096298b6b46b68e4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:45.303 INFO 2458 --- [alarm-detect-4,41befb36a819b0f8,096298b6b46b68e4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:17:45.338 INFO 2458 --- [alarm-detect-19,41befb36a819b0f8,aeffd5c55fbc3c95] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535035 ~ 1761535055 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535050, value=73.599)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535050, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535050=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:17:46.332 INFO 2458 --- [ocp-schedules-12,d1369145b984dfde,13edb00d8a82ab30] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:46.547 WARN 2458 --- [ocp-async-10,8e7d41965eb86cf2,c668db078bdb5599] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:46.547 INFO 2458 --- [ocp-async-8,aad7dfffd469a0f8,d79f153991fca94c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:47.042 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:17:47.065 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:47.068 INFO 2458 --- [ocp-schedules-4,6dca7ae49cae07dc,85bbb843c93c1272] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:17:47.069 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:47.077 INFO 2458 --- [ocp-schedules-4,6dca7ae49cae07dc,85bbb843c93c1272] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:47.077 INFO 2458 --- [ocp-schedules-4,6dca7ae49cae07dc,85bbb843c93c1272] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:17:47.121 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:47.124 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:47.173 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:47.176 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:47.222 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:47.225 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:47.277 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:47.279 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:47.333 INFO 2458 --- [ocp-schedules-3,c596007aa7f6a051,156eebb1f6709e57] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:17:47.707 INFO 2458 --- [prometheus-minute-schedule-1,ec412babfb09d6d8,752dab5fe8140a3c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:47.714 INFO 2458 --- [prometheus-minute-schedule-1,ec412babfb09d6d8,752dab5fe8140a3c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:48.211 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:48.212 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:48.367 INFO 2458 --- [ocp-schedules-8,65c42f74807edf31,41268d6e81974fb2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:17:48.367 INFO 2458 --- [ocp-schedules-8,65c42f74807edf31,41268d6e81974fb2] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:17:48.375 INFO 2458 --- [ocp-schedules-8,65c42f74807edf31,41268d6e81974fb2] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:17:48.375 INFO 2458 --- [ocp-schedules-8,65c42f74807edf31,41268d6e81974fb2] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:17:48.519 INFO 2458 --- [ocp-async-8,ca868a7681f3e1e2,db14491ada818b04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:48.547 INFO 2458 --- [ocp-async-7,5ccff9fd06dca1f7,fcbbdfb9aca27901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:48.547 WARN 2458 --- [ocp-async-1,a4d5d85069805ae7,602ca69124e65f00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:48.613 INFO 2458 --- [ocp-schedules-20,1fb31a85a8bb1bf7,dcf7bbfb3a1b5d39] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:17:48.613+08:00, clusters=[1, 2] 2025-10-27 11:17:48.633 INFO 2458 --- [ocp-schedules-20,1fb31a85a8bb1bf7,dcf7bbfb3a1b5d39] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:17:48.637 INFO 2458 --- [ocp-schedules-20,1fb31a85a8bb1bf7,dcf7bbfb3a1b5d39] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:17:50.547 WARN 2458 --- [ocp-async-6,56a4b813a1ea944a,ae519b34f00e45ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:50.547 INFO 2458 --- [ocp-async-7,e209aff33a4422dc,ada5fb6c811092b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:50.707 INFO 2458 --- [prometheus-minute-schedule-1,07ae1cd6089abe5e,b3ac64e8464c0d04] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:50.715 INFO 2458 --- [prometheus-minute-schedule-1,07ae1cd6089abe5e,b3ac64e8464c0d04] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:51.212 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:51.213 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:51.409 INFO 2458 --- [ocp-schedules-8,c68cfad0c215614a,deeff31362adf1be] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:51.409 INFO 2458 --- [ocp-schedules-19,e83c7d96cc744726,7b5803a762107f56] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:51.410 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:17:51.410 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:17:51.410 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:17:51.410 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:17:51.410 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:17:51.411 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:17:51.411 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:17:51.411 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:17:51.411 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:17:51.411 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:17:51.424 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:51.425 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:51.426 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:51.426 INFO 2458 --- [ocp-schedules-28,3f6cda6445c563af,cb76d827eea03ee4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:17:51.427 INFO 2458 --- [host-check-clockdiff1,3f6cda6445c563af,c890788e6afcae21] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:17:51.427 INFO 2458 --- [host-check-clockdiff2,3f6cda6445c563af,4060d1d3e050d0a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:17:51.427 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:51.428 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:51.428 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:51.428 INFO 2458 --- [host-check-clockdiff3,3f6cda6445c563af,8e673b8bf8106af2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:17:51.429 INFO 2458 --- [host-check-clockdiff4,3f6cda6445c563af,dc5678aa0b253e32] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:17:51.429 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:51.429 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:51.430 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:51.430 INFO 2458 --- [host-check-clockdiff5,3f6cda6445c563af,d1bc3d6febfee678] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:17:51.430 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:51.431 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:51.431 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:51.431 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:51.432 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:51.432 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:51.432 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:51.434 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:51.435 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:51.436 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:51.442 INFO 2458 --- [ocp-schedules-30,b1da839888ba0d37,160f0e67b8072da7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:17:51.458 INFO 2458 --- [host-check-clockdiff1,3f6cda6445c563af,c890788e6afcae21] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535071 0 0 , error= 2025-10-27 11:17:51.459 INFO 2458 --- [host-check-clockdiff3,3f6cda6445c563af,8e673b8bf8106af2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535071 -1 -1 , error= 2025-10-27 11:17:51.459 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:51.459 INFO 2458 --- [host-check-clockdiff4,3f6cda6445c563af,dc5678aa0b253e32] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535071 -1 -1 , error= 2025-10-27 11:17:51.469 INFO 2458 --- [host-check-clockdiff2,3f6cda6445c563af,4060d1d3e050d0a2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535071 -1 -1 , error= 2025-10-27 11:17:51.474 INFO 2458 --- [host-check-clockdiff5,3f6cda6445c563af,d1bc3d6febfee678] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535071 0 0 , error= 2025-10-27 11:17:51.509 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:17:51.509 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:17:51.509 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:17:51.510 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:17:51.510 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:17:51.510 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:17:51.511 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:17:51.510505623+08:00 2025-10-27 11:17:51.511 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:17:51.510403311+08:00 2025-10-27 11:17:51.512 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:17:51.511+08:00 2025-10-27 11:17:51.512 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:17:51.511996661+08:00 2025-10-27 11:17:51.512 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:17:51.511+08:00 2025-10-27 11:17:51.512 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:17:51.512+08:00 2025-10-27 11:17:51.513 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:17:51.513 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:17:51.513 INFO 2458 --- [host-check-agent-status2,e83c7d96cc744726,9b91f1932881408a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status3,e83c7d96cc744726,761d854140923ea0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:17:51.514 INFO 2458 --- [host-check-agent-status1,e83c7d96cc744726,7a0a73712402d64d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:17:51.515 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:17:51.515 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:17:51.515114258+08:00 2025-10-27 11:17:51.515 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:17:51.516 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:17:51.515+08:00 2025-10-27 11:17:51.517 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:17:51.518 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:17:51.518 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:17:51.518 INFO 2458 --- [host-check-agent-status4,e83c7d96cc744726,62eb53b1f35fede1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:17:51.521 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:17:51.521 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:17:51.521 INFO 2458 --- [host-check-agent-status5,e83c7d96cc744726,ca67bb5692dec020] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:17:51.527 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:17:51.527 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:17:51.511+08:00 and status AVAILABLE 2025-10-27 11:17:51.528 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:17:51.528093765+08:00 2025-10-27 11:17:51.528 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:17:51.528+08:00 2025-10-27 11:17:51.528 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:17:51.511+08:00 and status AVAILABLE 2025-10-27 11:17:51.529 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:17:51.515+08:00 and status AVAILABLE 2025-10-27 11:17:51.530 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:17:51.512+08:00 and status AVAILABLE 2025-10-27 11:17:51.532 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:17:51.534 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:17:51.536 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:17:51.528+08:00 and status AVAILABLE 2025-10-27 11:17:51.537 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:17:51.537 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:17:51.540 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:17:51.545 INFO 2458 --- [ocp-schedules-22,02cbf16fb26b2d3e,8e6b2676a1d5e77f] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:17:51.549 INFO 2458 --- [ocp-schedules-7,da5c32a0cc1a81db,e0159a4a3b9bf433] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:17:51.550 INFO 2458 --- [ocp-schedules-22,02cbf16fb26b2d3e,8e6b2676a1d5e77f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:51.552 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:17:51.553 INFO 2458 --- [ocp-schedules-7,da5c32a0cc1a81db,e0159a4a3b9bf433] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:51.553 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:17:51.554 INFO 2458 --- [ocp-schedules-13,89d3f63fa2fb9e6e,a269b302f71a59b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:17:51.554 INFO 2458 --- [ocp-schedules-7,da5c32a0cc1a81db,e0159a4a3b9bf433] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:17:51.554 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:17:51.555 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:17:51.556 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:17:51.557 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:51.557 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:17:51.558 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:51.560 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:51.560 INFO 2458 --- [ocp-schedules-13,89d3f63fa2fb9e6e,a269b302f71a59b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:51.561 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:51.567 INFO 2458 --- [ocp-schedules-7,22e1f556a812dbd9,98cf52d1293bce06] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:17:51.568 INFO 2458 --- [ocp-schedules-2,df3f7a0dc9e62bd1,47e0467e4e991ac9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:17:51.569 INFO 2458 --- [ocp-schedules-26,25afcbaec469f4c0,20cf5e87d2959e45] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:17:51.569 INFO 2458 --- [ocp-schedules-20,cb535b18f3d5720c,73f833087a6fe4af] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:17:51.576 INFO 2458 --- [ocp-schedules-26,25afcbaec469f4c0,20cf5e87d2959e45] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:51.578 INFO 2458 --- [ocp-schedules-2,df3f7a0dc9e62bd1,47e0467e4e991ac9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:17:51.579 INFO 2458 --- [ocp-schedules-20,cb535b18f3d5720c,73f833087a6fe4af] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:17:51.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:51.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:17:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.626 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:51.630 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.631 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:17:51.632 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:51.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:17:51.633 INFO 2458 --- [ocp-schedules-22,02cbf16fb26b2d3e,8e6b2676a1d5e77f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:17:51.634 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:51.639 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,bd5b8dd5551d8823,cc971d10ec4c308f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=bd5b8dd5551d8823, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:17:51.639 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,bd5b8dd5551d8823,cc971d10ec4c308f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=bd5b8dd5551d8823, duration=1 ms] 2025-10-27 11:17:51.640 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:51.640 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.641 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:51.644 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.644 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:17:51.648 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:51.649 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:17:51.651 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:17:51.655 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.658 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:17:51.660 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:17:51.664 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.665 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:17:51.670 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:17:51.673 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:17:51.695 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7c8f5a0bfb33a186,2cbc378130951794] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=7c8f5a0bfb33a186, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:17:51.695 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7c8f5a0bfb33a186,2cbc378130951794] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=7c8f5a0bfb33a186, duration=1 ms] 2025-10-27 11:17:51.698 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.698 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:17:51.704 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:17:51.704 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:17:51.715 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:17:51.716590+08:00, lastProcessTime=2025-10-27T11:17:41.716046+08:00 2025-10-27 11:17:51.718 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:17:51.719 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.722 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:17:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.722 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:51.723 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.723 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:17:51.725 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:17:51.727 INFO 2458 --- [ocp-schedules-26,25afcbaec469f4c0,20cf5e87d2959e45] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:17:51.728 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:51.728 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.729 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:17:51.738 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:17:51.752 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:51.753 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:51.762 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:51.768 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:51.769 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:17:51.772 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:17:51.786 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:17:51.793 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:17:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.801 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:17:51.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:51.802 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.803 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:51.804 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:17:51.804 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.805 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:17:51.807 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:17:51.808 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:51.814 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:17:51.833 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:17:51.851 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:51.857 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:17:51.858 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:51.860 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:51.870 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:17:51.873 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:17:51.875 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:17:51.878 INFO 2458 --- [ocp-schedules-11,f5ae04b59e23e749,6bad9aedb6dd5d36] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:17:51.878 INFO 2458 --- [ocp-schedules-11,f5ae04b59e23e749,6bad9aedb6dd5d36] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:17:51.879 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:17:51.889 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:51.893 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:51.911 INFO 2458 --- [host-check-host-status3,c68cfad0c215614a,e663a46eb3da21fc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:17:51.913 INFO 2458 --- [host-check-host-status5,c68cfad0c215614a,411fe57eae6e0098] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:17:51.914 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:17:51.919 INFO 2458 --- [host-check-host-status4,c68cfad0c215614a,14ceef0784783c9a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:17:51.920 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:17:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:51.932 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:17:51.932 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.933 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:51.935 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:17:51.935 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:17:51.936 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:17:51.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:51.951 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:17:51.958 INFO 2458 --- [host-check-host-status1,c68cfad0c215614a,49ff7fc7e012b2fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:17:51.961 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:17:51.964 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:17:51.976 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:17:51.976 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:51.976 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:51.998 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:17:51.716590+08:00 2025-10-27 11:17:52.014 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:17:52.035 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:17:52.036 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:17:52.052 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:52.052 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:52.064 INFO 2458 --- [host-check-host-status2,c68cfad0c215614a,e1bb971d89667e5c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:17:52.080 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:17:52.080 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.081 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:17:52.086 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.087 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:17:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:52.106 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:17:52.121 INFO 2458 --- [pool-overview-container-7,1d5b0b691817d7af,4324628547419e96] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.133 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:52.133 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:17:52.136 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:52.141 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.142 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:17:52.173 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:17:52.173 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:17:52.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:52.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:17:52.222 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:17:52.238 INFO 2458 --- [pool-overview-container-3,81b81bf5a4a576ea,a51fa298dd9a3750] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.272 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.275 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:17:52.277 INFO 2458 --- [pool-overview-container-4,5f844def5034b618,b8e6e2f8d3024a6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.280 INFO 2458 --- [pool-overview-container-4,5f844def5034b618,b8e6e2f8d3024a6e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:17:52.284 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:17:52.297 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:17:52.298 INFO 2458 --- [pool-12-thread-11,81b81bf5a4a576ea,8582e43c6f81135e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:17:52.304 INFO 2458 --- [pool-12-thread-12,81b81bf5a4a576ea,4bc8d088dfa34009] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:17:52.305 INFO 2458 --- [pool-12-thread-10,81b81bf5a4a576ea,479b84c1deafe540] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:17:52.305 INFO 2458 --- [pool-12-thread-9,81b81bf5a4a576ea,222f6942cc82b26c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:17:52.307 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:17:52.311 INFO 2458 --- [pool-12-thread-11,81b81bf5a4a576ea,8582e43c6f81135e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:17:52.311 INFO 2458 --- [pool-12-thread-11,81b81bf5a4a576ea,8582e43c6f81135e] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:17:52.311 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:17:52.311 INFO 2458 --- [ForkJoinPool-7-worker-14,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.312 INFO 2458 --- [ForkJoinPool-7-worker-7,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.312 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:17:52.312 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:17:52.312 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:17:52.312 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:17:52.329 INFO 2458 --- [pool-12-thread-9,81b81bf5a4a576ea,222f6942cc82b26c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:17:52.339 INFO 2458 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.342 INFO 2458 --- [ForkJoinPool-7-worker-0,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:17:52.352 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.352 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:17:52.353 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.354 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:17:52.363 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.363 INFO 2458 --- [pool-12-thread-9,81b81bf5a4a576ea,222f6942cc82b26c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:17:52.364 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:17:52.366 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:17:52.369 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.369 INFO 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:17:52.371 INFO 2458 --- [pool-12-thread-9,81b81bf5a4a576ea,222f6942cc82b26c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:17:52.382 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.384 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:17:52.393 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.396 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.399 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=n 2025-10-27 11:17:52.400 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:17:52.404 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.405 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.405 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.405 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:17:52.405 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:17:52.407 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:17:52.411 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 2 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.413 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.417 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:17:52.419 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:17:52.420 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.421 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:17:52.422 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.423 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:17:52.441 WARN 2458 --- [ForkJoinPool-7-worker-7,81b81bf5a4a576ea,831f457fc255d568] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:17:52.442 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:17:52.442 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.443 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:17:52.445 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:17:52.448 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:17:52.455 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.456 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:17:52, 2025-10-27 11:17:52] 2025-10-27 11:17:52.474 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.476 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:17:52.478 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:17:52.480 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.481 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:17:52.482 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.482 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.483 INFO 2458 --- [ForkJoinPool-7-worker-9,81b81bf5a4a576ea,eb41f647b397e9bf] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:17:52.485 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:17:52.487 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:17:52.491 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:17:52.491 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.493 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:17:52, 2025-10-27 11:17:52] 2025-10-27 11:17:52.494 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:17:52.495 INFO 2458 --- [ForkJoinPool-7-worker-14,81b81bf5a4a576ea,e5ced0d0f06d8949] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:17:52.495 INFO 2458 --- [pool-12-thread-10,81b81bf5a4a576ea,479b84c1deafe540] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:17:52.503 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:17:52.503 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.504 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.505 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:17:52.506 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.506 INFO 2458 --- [ForkJoinPool-7-worker-0,81b81bf5a4a576ea,838a7e32a4a377eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:17:52.507 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 2 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.507 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:52.513 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.519 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:17:52.542 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:52.542 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.543 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:17:52.547 INFO 2458 --- [pool-12-thread-12,81b81bf5a4a576ea,4bc8d088dfa34009] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:17:52.547 INFO 2458 --- [ocp-async-10,41dc5b4291d26cd9,2464c630da3c1a37] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:52.547 WARN 2458 --- [ocp-async-8,f14acc3c32db297e,3af09af527007450] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:52.559 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.560 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.561 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:17:52.565 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.566 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.573 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.574 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.576 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.577 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.578 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.579 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.581 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.584 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:17:52.584 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:17:52.584 INFO 2458 --- [ocp-schedules-13,89d3f63fa2fb9e6e,a269b302f71a59b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:17:52.634 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:17:52.634 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:17:52.638 INFO 2458 --- [refresh-tenant-resource-cache-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:17:52.694 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.695 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.697 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.698 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.699 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.700 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.701 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.702 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.703 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.709 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.709 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:17:52.712 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.717 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.719 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:11:52.087, end=2025-10-27T11:16:52.385} 2025-10-27 11:17:52.719 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 1 dubious sqls 2025-10-27 11:17:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:17:53.222 INFO 2458 --- [ocp-schedules-14,fa8ce61166647b1c,02c0a165b2bf77aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20379,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:17:53.222 INFO 2458 --- [ocp-schedules-14,fa8ce61166647b1c,02c0a165b2bf77aa] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:17:53.707 INFO 2458 --- [prometheus-minute-schedule-1,66c5d4d28df54023,748b6d2dfe0073fa] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:53.718 INFO 2458 --- [prometheus-minute-schedule-1,66c5d4d28df54023,748b6d2dfe0073fa] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:54.213 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:54.214 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:17:54.396 INFO 2458 --- [ocp-schedules-11,ac6ddc1a3ed76dfd,17a5f0518ee0f293] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:17:54.399 INFO 2458 --- [ocp-schedules-11,ac6ddc1a3ed76dfd,17a5f0518ee0f293] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:17:54.548 WARN 2458 --- [ocp-async-10,495dcb012e547b04,dec71f74eb4c11b3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:54.548 INFO 2458 --- [ocp-async-10,1ac08aa6ec1519f9,20f8fcfb471599ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:54.884 INFO 2458 --- [ocp-schedules-12,8149e500372bfa92,dd32c21ff21c145f] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:17:55.347 INFO 2458 --- [ocp-schedules-22,561ef77a2582ffa3,b9ca9612422933c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:17:55.392 WARN 2458 --- [scanner-112163952-1,561ef77a2582ffa3,979c97a4c3e10a16] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:17:55.479 INFO 2458 --- [alarm-detect-7,561ef77a2582ffa3,04009ef9a6af45e7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535050 ~ 1761535065 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535065, value=68.023)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535065, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535065=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:17:56.339 INFO 2458 --- [ocp-schedules-19,a7e6eac53c59f6ba,3701164ce5f11bf2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:17:56.548 WARN 2458 --- [ocp-async-10,e29ea3c6cf806582,b8124e4ee6ae0c59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:56.548 INFO 2458 --- [ocp-async-3,200622af277224e1,ba601e8f15d8ccb7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:56.707 INFO 2458 --- [prometheus-minute-schedule-1,75ef1801f8b33d2d,2b58905e29438370] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:56.716 INFO 2458 --- [prometheus-minute-schedule-1,75ef1801f8b33d2d,2b58905e29438370] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:17:57.214 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:17:57.215 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:17:57.947 INFO 2458 --- [ocp-schedules-14,d9035b88819ff665,71c8bc0c0a13a11e] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:17:57.953 INFO 2458 --- [ocp-schedules-14,d9035b88819ff665,71c8bc0c0a13a11e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:17:57.998 INFO 2458 --- [ocp-schedules-14,d9035b88819ff665,71c8bc0c0a13a11e] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:17:58.519 INFO 2458 --- [ocp-async-3,73d0f2672fa9956a,58d40ce740414e08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:17:58.548 WARN 2458 --- [ocp-async-1,fa9a16624c4155a9,8f7b610074913f51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:17:58.548 INFO 2458 --- [ocp-async-8,a89323083264abff,e8846d27cad41035] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:17:59.313 INFO 2458 --- [ocp-schedules-23,294ccace51df5c5e,7bc5ebc4f0212321] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:17:59.707 INFO 2458 --- [prometheus-minute-schedule-1,4d0cdd4cd226e035,77295cdee5962b68] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535080, interval=60 2025-10-27 11:17:59.717 INFO 2458 --- [prometheus-minute-schedule-1,4d0cdd4cd226e035,77295cdee5962b68] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:00.215 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:00.216 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:00.548 INFO 2458 --- [ocp-async-8,bb84df70c94cc5d4,ceb9cf23819121eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:00.548 WARN 2458 --- [ocp-async-5,f2c5b52a3b50967e,08a503c2b7970bfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:01.714867+08:00, lastProcessTime=2025-10-27T11:17:51.716590+08:00 2025-10-27 11:18:02.265 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:01.714867+08:00 2025-10-27 11:18:02.548 INFO 2458 --- [ocp-async-8,9197daa0ec1ee8d9,fbcff7615ed2c746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:02.548 WARN 2458 --- [ocp-async-7,b680a384c7b8262b,17f1062be94cfdc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:02.707 INFO 2458 --- [prometheus-minute-schedule-1,268833db70c4adb2,4242a63378019c49] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:02.728 INFO 2458 --- [prometheus-minute-schedule-1,268833db70c4adb2,4242a63378019c49] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:03.216 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:03.217 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:03.255 INFO 2458 --- [ocp-schedules-25,476a3579cc376d3c,219025672c9334a0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:18:03.276 INFO 2458 --- [ocp-schedules-25,476a3579cc376d3c,219025672c9334a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:03.334 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:03.334 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:18:03.342 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:03.343 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:18:03.363 INFO 2458 --- [ocp-schedules-25,476a3579cc376d3c,219025672c9334a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:18:04.407 INFO 2458 --- [ocp-schedules-2,56b47947801af7f9,bbb45dc1b85dccca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:04.411 INFO 2458 --- [ocp-schedules-2,56b47947801af7f9,bbb45dc1b85dccca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:04.475 INFO 2458 --- [ocp-schedules-15,257827bf5d545224,7c5f61a7cfb267e5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:18:04.483 INFO 2458 --- [ocp-schedules-15,257827bf5d545224,7c5f61a7cfb267e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:04.548 INFO 2458 --- [ocp-async-1,da9f12e9da0c376a,b7693c747647c4e6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:04.548 WARN 2458 --- [ocp-async-6,48ce665c2c20b8d8,869b3879a219f8ef] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:04.572 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:18:04.573 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.576 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:18:04.589 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.590 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.617 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.617 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.641 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:18:04.641 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.645 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:18:04.647 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.647 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.649 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.650 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.663 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.664 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.676 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.676 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.689 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.690 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.703 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.703 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.720 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:18:04.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.724 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.737 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.738 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.738 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.739 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.740 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.741 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.741 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.742 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.742 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.743 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.744 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.745 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.745 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:18:04.751 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.751 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:18:04.751 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.776 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.777 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:18:04.779 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.779 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:18:04.793 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:04.794 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.794 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:18:04.805 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:04.805 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:18:04.805 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:04.805 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:18:04.808 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:18:04.808 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:18:04.810 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.810 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:18:04.839 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.839 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:18:04.841 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.842 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:18:04.869 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.869 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:18:04.895 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.896 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:18:04.942 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.943 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.944 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.945 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.945 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.946 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.947 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.947 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.948 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.949 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.949 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.950 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.952 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.953 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.953 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.954 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.955 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.955 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:18:04.956 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.957 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:18:04.957 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:04.958 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:18:04.958 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:18:04.961 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:18:05.044 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:05.045 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:05.045 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:18:05.070 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:18:05.074 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:18:05.074 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:18:05.074 INFO 2458 --- [ocp-schedules-15,257827bf5d545224,7c5f61a7cfb267e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:18:05.488 INFO 2458 --- [ocp-schedules-20,4a27087700027d28,dd6df5667381a5f9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:05.533 WARN 2458 --- [scanner-112164107-1,4a27087700027d28,c0fe8067d7eeefc9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:05.561 INFO 2458 --- [alarm-detect-17,4a27087700027d28,e4961c84bea351c5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:05.565 INFO 2458 --- [alarm-detect-17,4a27087700027d28,e4961c84bea351c5] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:05.572 INFO 2458 --- [alarm-detect-10,4a27087700027d28,73de522772f4697b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:05.573 INFO 2458 --- [alarm-detect-14,4a27087700027d28,02186e5799cba5cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:05.578 INFO 2458 --- [alarm-detect-10,4a27087700027d28,73de522772f4697b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:05.578 INFO 2458 --- [alarm-detect-14,4a27087700027d28,02186e5799cba5cb] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:05.632 INFO 2458 --- [alarm-detect-7,4a27087700027d28,153e865f9e7865ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:05.637 INFO 2458 --- [alarm-detect-7,4a27087700027d28,153e865f9e7865ef] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:05.707 INFO 2458 --- [prometheus-minute-schedule-1,c35a76ffa534916f,e98e461360248c5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:05.714 INFO 2458 --- [prometheus-minute-schedule-1,c35a76ffa534916f,e98e461360248c5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:18:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:06.003 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:18:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:18:06.066 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:18:06.117 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:18:06.138 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:18:06.138 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:18:06.138 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:18:06.138 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:18:06.140 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:18:06.140 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:18:06.141 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:18:06.141 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:18:06.141 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:18:06.141 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:18:06.141 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:06.142 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:18:06.144 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:06.144 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:18:06.211 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:15:30, start=2025-10-27T11:12:30, end=2025-10-27T11:15:00, updates=264, granule=120000ms, delay=180000ms 2025-10-27 11:18:06.217 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:06.218 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:06.221 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:15:30, target=2025-10-27T11:15:00, granule=120000ms, delay=30000ms) 2025-10-27 11:18:06.221 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:15:00, target=2025-10-27T11:02:30, granule=600000ms, delay=750000ms) 2025-10-27 11:18:06.221 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=217ms, period=120000ms 2025-10-27 11:18:06.348 INFO 2458 --- [ocp-schedules-14,af241c529e1dfe5e,24d5d474fb9dca8f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:06.548 WARN 2458 --- [ocp-async-10,32d71b6701899a2c,efbaba6e85e2667a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:06.549 INFO 2458 --- [ocp-async-9,bb4c2a6bf3b6aa9e,377ce12f14c291b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:07.080 INFO 2458 --- [ocp-schedules-18,b085ad438e7af075,262ae852a9ad68f7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:18:07.088 INFO 2458 --- [ocp-schedules-18,b085ad438e7af075,262ae852a9ad68f7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:07.088 INFO 2458 --- [ocp-schedules-18,b085ad438e7af075,262ae852a9ad68f7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:08.519 INFO 2458 --- [ocp-async-7,8f7e9143412c355f,088c4f333fbb2731] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:08.549 WARN 2458 --- [ocp-async-3,147213487fdaf3e2,fbc40a0a02b0899c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:08.549 INFO 2458 --- [ocp-async-8,bc8212877e16b8c3,1ccac24522d8db73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:08.707 INFO 2458 --- [prometheus-minute-schedule-1,9dd84b6a2eae767f,962fbc62b8e995ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:08.718 INFO 2458 --- [prometheus-minute-schedule-1,9dd84b6a2eae767f,962fbc62b8e995ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:09.218 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:09.220 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:10.028 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:18:10.028 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:18:10.091 INFO 2458 --- [Check obproxy1,c67b33dfd10bb82e,eb8e602557f53698] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:18:10.091 INFO 2458 --- [Check obproxy1,c67b33dfd10bb82e,eb8e602557f53698] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:18:10.093 INFO 2458 --- [Collect obproxy1,753842110680a163,99fd09a99e15c35c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:18:10.093 INFO 2458 --- [Collect obproxy1,753842110680a163,99fd09a99e15c35c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:18:10.121 INFO 2458 --- [Check obproxy1,c67b33dfd10bb82e,eb8e602557f53698] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:18:10.125 INFO 2458 --- [Collect obproxy1,753842110680a163,99fd09a99e15c35c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:18:10.125 INFO 2458 --- [Check obproxy1,c67b33dfd10bb82e,eb8e602557f53698] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:18:10.125 INFO 2458 --- [Check obproxy1,c67b33dfd10bb82e,eb8e602557f53698] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:18:10.128 INFO 2458 --- [Collect obproxy1,753842110680a163,99fd09a99e15c35c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:18:10.130 INFO 2458 --- [Collect obproxy1,753842110680a163,99fd09a99e15c35c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:18:10.185 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:18:10.549 WARN 2458 --- [ocp-async-4,2b5605988621d87d,316abcdb4c10fefc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:10.549 INFO 2458 --- [ocp-async-1,16894a185cc0ee75,67eb012bdde5c02a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:10.991 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535020, nodeCount=294, affectRows=294 2025-10-27 11:18:11.012 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535020, nodeCount=100, affectRows=100 2025-10-27 11:18:11.042 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535020, nodeCount=98, affectRows=98 2025-10-27 11:18:11.120 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=1000, affectRows=1000 2025-10-27 11:18:11.125 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=994, validCount=992 2025-10-27 11:18:11.188 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=1000, affectRows=1000 2025-10-27 11:18:11.192 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=999, validCount=995 2025-10-27 11:18:11.211 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587936) Duplicate entry '835014129065861120-1761534960' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587936) Duplicate entry '835014129065861120-1761534960' for key 'PRIMARY' 2025-10-27 11:18:11.707 INFO 2458 --- [prometheus-minute-schedule-1,05de2286d1bc47cf,4ceff4b1ff751246] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:11.714 INFO 2458 --- [prometheus-minute-schedule-1,05de2286d1bc47cf,4ceff4b1ff751246] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:11.713012+08:00, lastProcessTime=2025-10-27T11:18:01.714867+08:00 2025-10-27 11:18:11.943 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:11.713012+08:00 2025-10-27 11:18:12.220 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:12.221 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:12.549 INFO 2458 --- [ocp-async-10,d988b23608fc7613,f9e03dce9915138c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:12.549 WARN 2458 --- [ocp-async-9,106f6a10370337ab,117261b159ebd39c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:12.743 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505650) Duplicate entry '835370603008958464-1761534960' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505650) Duplicate entry '835370603008958464-1761534960' for key 'PRIMARY' 2025-10-27 11:18:12.768 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=159, affectRows=159 2025-10-27 11:18:12.769 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=158, validCount=158 2025-10-27 11:18:12.781 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=1000, affectRows=1000 2025-10-27 11:18:12.786 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=1000, validCount=996 2025-10-27 11:18:13.810 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535020, nodeCount=325, affectRows=325 2025-10-27 11:18:14.419 INFO 2458 --- [ocp-schedules-8,7bbec221d6063882,d10fd4de60351b79] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:14.423 INFO 2458 --- [ocp-schedules-8,7bbec221d6063882,d10fd4de60351b79] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:14.549 INFO 2458 --- [ocp-async-3,60103a1e1abca7a2,29eb3641f806bdd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:14.549 WARN 2458 --- [ocp-async-2,9258cda40b94cc06,5ffa8b7b27a886f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:14.707 INFO 2458 --- [prometheus-minute-schedule-1,d733324f3bc1d23f,a3f02338ebf77882] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:14.714 INFO 2458 --- [prometheus-minute-schedule-1,d733324f3bc1d23f,a3f02338ebf77882] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:15.048 INFO 2458 --- [ocp-schedules-19,b04f560f93c0396c,6ce7a58db2c94c46] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:18:15.048+08:00, clusters=[1, 2] 2025-10-27 11:18:15.051 INFO 2458 --- [ocp-schedules-19,b04f560f93c0396c,6ce7a58db2c94c46] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:18:15.052 INFO 2458 --- [ocp-schedules-19,b04f560f93c0396c,6ce7a58db2c94c46] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:18:15.221 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:15.222 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:15.660 INFO 2458 --- [ocp-schedules-13,1fc57806569eb923,c200ad66e9b1ed9a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:15.686 WARN 2458 --- [scanner-112165912-1,1fc57806569eb923,6ce1c5732cb3b0ea] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:15.780 INFO 2458 --- [alarm-detect-19,1fc57806569eb923,d4bc695cc08da02a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535065 ~ 1761535085 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535080, value=56.221)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535080, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535080=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:18:15.829 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=1000, affectRows=1000 2025-10-27 11:18:15.833 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=996, validCount=994 2025-10-27 11:18:15.847 INFO 2458 --- [ocp-schedules-16,659ad4148be2c2bd,124ac7fba64a3a50] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:18:15.847 INFO 2458 --- [ocp-schedules-16,659ad4148be2c2bd,124ac7fba64a3a50] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:18:15.921 INFO 2458 --- [Collect all tenants major compactions1,659ad4148be2c2bd,a49149a964324a61] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:18:15.922 INFO 2458 --- [Collect all tenants major compactions2,659ad4148be2c2bd,e709eb46cf8d925e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:18:15.924 INFO 2458 --- [Collect all tenants major compactions3,659ad4148be2c2bd,eb76f6aeb0395786] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:18:15.930 INFO 2458 --- [Collect all tenants major compactions5,659ad4148be2c2bd,3c0f8e02eddec6fd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:18:15.930 INFO 2458 --- [Collect all tenants major compactions4,659ad4148be2c2bd,0b4c74b4aab8d926] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:18:15.930 INFO 2458 --- [Collect all tenants major compactions6,659ad4148be2c2bd,5c8c71f12a3c6af7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:18:16.000 INFO 2458 --- [Collect all tenants major compactions6,659ad4148be2c2bd,5c8c71f12a3c6af7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.002 INFO 2458 --- [Collect all tenants major compactions6,659ad4148be2c2bd,5c8c71f12a3c6af7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:18:16.004 INFO 2458 --- [Collect all tenants major compactions5,659ad4148be2c2bd,3c0f8e02eddec6fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.006 INFO 2458 --- [Collect all tenants major compactions5,659ad4148be2c2bd,3c0f8e02eddec6fd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:18:16.017 INFO 2458 --- [Collect all tenants major compactions6,659ad4148be2c2bd,5c8c71f12a3c6af7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:18:16.018 INFO 2458 --- [Collect all tenants major compactions5,659ad4148be2c2bd,3c0f8e02eddec6fd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:18:16.034 INFO 2458 --- [Collect all tenants major compactions1,659ad4148be2c2bd,a49149a964324a61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.036 INFO 2458 --- [Collect all tenants major compactions1,659ad4148be2c2bd,a49149a964324a61] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:18:16.038 INFO 2458 --- [Collect all tenants major compactions2,659ad4148be2c2bd,e709eb46cf8d925e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.039 INFO 2458 --- [Collect all tenants major compactions2,659ad4148be2c2bd,e709eb46cf8d925e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:18:16.043 INFO 2458 --- [Collect all tenants major compactions1,659ad4148be2c2bd,a49149a964324a61] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:18:16.044 INFO 2458 --- [Collect all tenants major compactions2,659ad4148be2c2bd,e709eb46cf8d925e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:18:16.046 INFO 2458 --- [Collect all tenants major compactions4,659ad4148be2c2bd,0b4c74b4aab8d926] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.046 INFO 2458 --- [Collect all tenants major compactions3,659ad4148be2c2bd,eb76f6aeb0395786] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:16.047 INFO 2458 --- [Collect all tenants major compactions4,659ad4148be2c2bd,0b4c74b4aab8d926] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:18:16.047 INFO 2458 --- [Collect all tenants major compactions3,659ad4148be2c2bd,eb76f6aeb0395786] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:18:16.051 INFO 2458 --- [Collect all tenants major compactions3,659ad4148be2c2bd,eb76f6aeb0395786] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:18:16.051 INFO 2458 --- [Collect all tenants major compactions4,659ad4148be2c2bd,0b4c74b4aab8d926] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:18:16.052 INFO 2458 --- [ocp-schedules-16,659ad4148be2c2bd,124ac7fba64a3a50] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:18:16.355 INFO 2458 --- [ocp-schedules-21,4180f3dbda54c679,863f7372b806ef25] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:16.549 INFO 2458 --- [ocp-async-8,a67e62fcf4f7fbf2,2d1aec280dbb380e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:16.549 WARN 2458 --- [ocp-async-4,0fe6fc05eb0e79d2,2bb187e5ebc332a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:17.172 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535020, nodeCount=940, affectRows=940 2025-10-27 11:18:17.176 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535020, nodeCount=936, validCount=921 2025-10-27 11:18:17.263 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258} 2025-10-27 11:18:17.270 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterVi 2025-10-27 11:18:17.270 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.277 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.289 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:18:17.289 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.359 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3004196468ee31b7,a9dbed8d1573234c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=3004196468ee31b7, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:18:17.359 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3004196468ee31b7,a9dbed8d1573234c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=3004196468ee31b7, duration=1 ms] 2025-10-27 11:18:17.707 INFO 2458 --- [prometheus-minute-schedule-1,3babbb374e79eedf,5ea67fd6e7d679f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:17.714 INFO 2458 --- [prometheus-minute-schedule-1,3babbb374e79eedf,5ea67fd6e7d679f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:17.792 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448697792000, endTimeUs=1761534977258000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:18:17.792Z, endTime=2025-10-27T03:16:17.258Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:18:17.803 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:18:17.803 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:18:17.825 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:18:17.825 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534977258000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:18:17.840 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:18:17.840 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:18:17.864 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:18:17.864 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761534977258000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:18:17.868 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:18:17.868 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:18:17.792, end=2025-10-27T11:16:17.258}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:18:17.869 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:18:17.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.873 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:18:17.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.876 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.877 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.878 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:18:17.878 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.880 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.883 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.883 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:18:17.883 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.888 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:18:17.890 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:14:16.162, end=2025-10-27T11:16:17.258},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:18:17.890 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:18:17.890 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:18:17.892 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:18:18.222 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:18.223 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:18.519 INFO 2458 --- [ocp-async-1,52d3f2a9260f5768,230b7e40f20d36cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:18.524 INFO 2458 --- [ocp-schedules-26,def6d5d56740cd37,05ea9b2535d7ba35] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:18:18.524 INFO 2458 --- [ocp-schedules-26,def6d5d56740cd37,05ea9b2535d7ba35] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:18:18.534 INFO 2458 --- [ocp-schedules-26,def6d5d56740cd37,05ea9b2535d7ba35] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:18:18.534 INFO 2458 --- [ocp-schedules-26,def6d5d56740cd37,05ea9b2535d7ba35] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:18:18.549 WARN 2458 --- [ocp-async-9,17933ebb22ea9dce,4e2333e8bd74c391] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:18.549 INFO 2458 --- [ocp-async-10,632670f24948bc0d,693d09d238e42972] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:18.899 INFO 2458 --- [ocp-schedules-11,f101f2cbe57d4742,25e4fba7a9b5f669] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:18.899 INFO 2458 --- [ocp-schedules-11,f101f2cbe57d4742,25e4fba7a9b5f669] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:18:19.808 INFO 2458 --- [ocp-schedules-19,64a7ceaec8fcab30,13876115f747c6b5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:18:20.549 WARN 2458 --- [ocp-async-2,3a499a41302e3953,d2354f9f0ef22d1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:20.550 INFO 2458 --- [ocp-async-2,9f2758f4336a0004,842f84c9f7704a60] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:20.707 INFO 2458 --- [prometheus-minute-schedule-1,17a38f2971bb2279,5621403aeb5cd925] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:20.715 INFO 2458 --- [prometheus-minute-schedule-1,17a38f2971bb2279,5621403aeb5cd925] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:21.223 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:21.223 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:21.409 INFO 2458 --- [ocp-schedules-9,d36593eb1fdba5e3,11b8d994293b2bf9] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:21.409 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:18:21.410 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:18:21.410 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:18:21.410 INFO 2458 --- [ocp-schedules-4,d052cf71f61adcec,10f1479d00a779bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:21.410 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:18:21.410 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:18:21.410 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:18:21.411 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:18:21.411 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:18:21.412 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:18:21.412 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:18:21.424 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:21.424 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:21.424 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:21.426 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:21.428 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:21.429 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:21.429 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:21.429 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:21.429 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:21.430 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:21.430 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:21.430 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:21.432 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:21.432 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:21.433 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:21.433 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:21.433 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:21.435 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:21.435 INFO 2458 --- [ocp-schedules-10,3901a882d6f21ae2,8cfaace51440d242] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:21.435 INFO 2458 --- [host-check-clockdiff1,3901a882d6f21ae2,41161a1730896269] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:18:21.436 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:21.437 INFO 2458 --- [host-check-clockdiff3,3901a882d6f21ae2,c280d6d2b08d2b6f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:18:21.437 INFO 2458 --- [host-check-clockdiff2,3901a882d6f21ae2,4e14c1702bd0212f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:18:21.437 INFO 2458 --- [host-check-clockdiff4,3901a882d6f21ae2,72436bd2ec48be56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:18:21.437 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:21.437 INFO 2458 --- [host-check-clockdiff5,3901a882d6f21ae2,f2c32eab9d8b0881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:18:21.442 INFO 2458 --- [ocp-schedules-21,494587ea5d6d9128,bd3a9bc3be9e6d4e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:18:21.494 INFO 2458 --- [host-check-clockdiff2,3901a882d6f21ae2,4e14c1702bd0212f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535101 -2 -2 , error= 2025-10-27 11:18:21.495 INFO 2458 --- [host-check-clockdiff3,3901a882d6f21ae2,c280d6d2b08d2b6f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535101 -1 -1 , error= 2025-10-27 11:18:21.507 INFO 2458 --- [host-check-clockdiff4,3901a882d6f21ae2,72436bd2ec48be56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535101 -1 -1 , error= 2025-10-27 11:18:21.546 INFO 2458 --- [ocp-schedules-23,5ff1cc9a09fa04c6,d15a0eb628482c41] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:18:21.549 INFO 2458 --- [ocp-schedules-5,1f9960a23f89314c,f37c569014fdc911] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:18:21.554 INFO 2458 --- [ocp-schedules-2,c994ec920b9f949c,22e0f936b5923d91] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:18:21.556 INFO 2458 --- [ocp-schedules-5,1f9960a23f89314c,f37c569014fdc911] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:21.556 INFO 2458 --- [ocp-schedules-5,1f9960a23f89314c,f37c569014fdc911] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:18:21.559 INFO 2458 --- [ocp-schedules-2,c994ec920b9f949c,22e0f936b5923d91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:21.567 INFO 2458 --- [ocp-schedules-29,db8b00b4ef330ee2,d3ef5c173335f03b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:18:21.575 INFO 2458 --- [ocp-schedules-23,5ff1cc9a09fa04c6,d15a0eb628482c41] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:21.620 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:18:21.622 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:18:21.621506661+08:00 2025-10-27 11:18:21.623 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:18:21.623+08:00 2025-10-27 11:18:21.625 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:18:21.625 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:18:21.628 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:18:21.628 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:18:21.628 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:18:21.628 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:18:21.628 INFO 2458 --- [host-check-agent-status2,d052cf71f61adcec,147364d744422570] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:18:21.629 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:18:21.628764399+08:00 2025-10-27 11:18:21.630 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:18:21.630 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:18:21.630 INFO 2458 --- [host-check-agent-status5,d052cf71f61adcec,26ae42c95f9dbcad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:18:21.631 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:18:21.629+08:00 2025-10-27 11:18:21.632 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:18:21.632 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:18:21.632 INFO 2458 --- [host-check-agent-status1,d052cf71f61adcec,19c7034c5d8b95fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:18:21.635 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:18:21.623+08:00 and status AVAILABLE 2025-10-27 11:18:21.639 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:18:21.639 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:18:21.640 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:18:21.629+08:00 and status AVAILABLE 2025-10-27 11:18:21.641 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:18:21.640722839+08:00 2025-10-27 11:18:21.642 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.642 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:18:21.641+08:00 2025-10-27 11:18:21.642 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:21.643 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:18:21.645 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:18:21.647 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.648 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:21.656 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:18:21.641+08:00 and status AVAILABLE 2025-10-27 11:18:21.657 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:18:21.660 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:18:21.660 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:18:21.660 INFO 2458 --- [host-check-agent-status3,d052cf71f61adcec,2e907f070b989ef9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:18:21.660 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.661 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:18:21.661 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:21.662 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:18:21.663 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:18:21.663372231+08:00 2025-10-27 11:18:21.664 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:18:21.663+08:00 2025-10-27 11:18:21.666 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:18:21.667 INFO 2458 --- [ocp-schedules-23,5ff1cc9a09fa04c6,d15a0eb628482c41] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:18:21.673 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:18:21.663+08:00 and status AVAILABLE 2025-10-27 11:18:21.678 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:18:21.686 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:18:21.686 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:18:21.686 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:18:21.686718602+08:00 2025-10-27 11:18:21.689 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:18:21.686+08:00 2025-10-27 11:18:21.690 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:21.691 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:21.691 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:18:21.691 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:18:21.691 INFO 2458 --- [host-check-agent-status4,d052cf71f61adcec,da05cdac4ae51c7a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:18:21.693 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:21.694 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:21.698 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:18:21.686+08:00 and status AVAILABLE 2025-10-27 11:18:21.700 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:21.700 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:21.702 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:21.703 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:18:21.703 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:21.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:21.713170+08:00, lastProcessTime=2025-10-27T11:18:11.713012+08:00 2025-10-27 11:18:21.717 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.717 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:21.725 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:21.729 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:21.734 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:21.735 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.735 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:18:21.778 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:21.778 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:21.798 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:21.799 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:21.805 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:18:21.811 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:18:21.813 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:21.814 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:18:21.820 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:18:21.821 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:21.822 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.822 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:18:21.826 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:21.826 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.827 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.830 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:18:21.831 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:18:21.832 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:21.851 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:18:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:21.853 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:18:21.855 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:18:21.856 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:18:21.861 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:18:21.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:21.892 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:21.893 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:21.893 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:21.894 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:21.894 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:21.909 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:18:21.911 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:21.911 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.912 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:21.912 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:18:21.913 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:18:21.913 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.913 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:18:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:21.915 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:21.916 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:18:21.917 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:18:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:21.922 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:18:21.931 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:18:21.941 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:21.944 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:21.944 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:18:21.950 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:18:21.951 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:18:21.952 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:18:21.956 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:18:21.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:21.973 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:21.979 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:21.980 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:21.986 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:21.990 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:21.993 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:21.993 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:18:21.999 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:18:22.005 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:18:22.006 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:21.713170+08:00 2025-10-27 11:18:22.007 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:18:22.030 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:18:22.031 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:18:22.034 INFO 2458 --- [host-check-host-status4,d36593eb1fdba5e3,86836f419e453590] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:18:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:22.047 INFO 2458 --- [host-check-host-status2,d36593eb1fdba5e3,659ab1f1c4f365d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:18:22.051 INFO 2458 --- [host-check-host-status3,d36593eb1fdba5e3,fa435d65eae044c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:18:22.052 INFO 2458 --- [host-check-host-status5,d36593eb1fdba5e3,37bb45258c20f3e0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:18:22.061 INFO 2458 --- [host-check-host-status1,d36593eb1fdba5e3,f72ea6351cb3188a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:18:22.075 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:18:22.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:18:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.079 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:18:22.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:22.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:22.095 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:18:22.120 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:22.121 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.121 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:18:22.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:22.123 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:22.143 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:18:22.143 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:18:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:22.233 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:22.234 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.234 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:22.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:18:22.248 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:18:22.258 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:22.261 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:22.307 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:18:22.383 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:18:22.383 INFO 2458 --- [host-check-clockdiff1,3901a882d6f21ae2,41161a1730896269] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535102 0 0 , error= 2025-10-27 11:18:22.386 INFO 2458 --- [host-check-clockdiff5,3901a882d6f21ae2,f2c32eab9d8b0881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535102 0 0 , error= 2025-10-27 11:18:22.386 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.389 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:18:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:18:22.425 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:18:22.472 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:22.473 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:22.473 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:18:22.513 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:18:22.513 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:18:22.513 INFO 2458 --- [ocp-schedules-2,c994ec920b9f949c,22e0f936b5923d91] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:18:22.550 WARN 2458 --- [ocp-async-6,7631d0fe214132ac,d1e7bfc639a5ac24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:22.550 INFO 2458 --- [ocp-async-6,2324a8e507cc91d3,86f6f75baf0fec21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:23.707 INFO 2458 --- [prometheus-minute-schedule-1,c5323bb5cae4d8cd,3f010be1abb0db9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:23.715 INFO 2458 --- [prometheus-minute-schedule-1,c5323bb5cae4d8cd,3f010be1abb0db9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:24.007 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : Deleted expired service: DistributedService{name=SQL_INDEX_DIAGNOSER, serverId=10.40.65.146:8081, expireInSeconds=30, createTime=2025-10-27T09:39:53+08:00, updateTime=2025-10-27T11:17:53+08:00, expireTime=null, isLeader=false} 2025-10-27 11:18:24.008 WARN 2458 --- [ocp-updater-0,,] c.o.o.c.d.service.DefaultServiceUpdater : No leader or more than one leader exists for service SQL_INDEX_DIAGNOSER, Incorrect result size: expected 1, actual 0 2025-10-27 11:18:24.224 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:24.224 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:24.431 INFO 2458 --- [ocp-schedules-27,b6569ad27f1f6394,baebe0bb712b721f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:24.434 INFO 2458 --- [ocp-schedules-27,b6569ad27f1f6394,baebe0bb712b721f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:24.550 WARN 2458 --- [ocp-async-9,0f210b0949cfab9c,a306c9622dee5e1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:24.550 INFO 2458 --- [ocp-async-5,a3213d4fe1c3a409,4a13eb7923d7d1d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:25.789 INFO 2458 --- [ocp-schedules-29,2d8c3cf42c335491,18b60ccdbf0b5537] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:25.804 WARN 2458 --- [scanner-112166077-1,2d8c3cf42c335491,2a35699e94a4073f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:25.829 INFO 2458 --- [alarm-detect-12,2d8c3cf42c335491,357a33f60ec3d128] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:25.832 INFO 2458 --- [alarm-detect-12,2d8c3cf42c335491,357a33f60ec3d128] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:25.849 INFO 2458 --- [alarm-detect-14,2d8c3cf42c335491,f991474e4021e155] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:25.854 INFO 2458 --- [alarm-detect-14,2d8c3cf42c335491,f991474e4021e155] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:25.855 INFO 2458 --- [alarm-detect-18,2d8c3cf42c335491,9fe6529aa3db4312] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:25.859 INFO 2458 --- [alarm-detect-18,2d8c3cf42c335491,9fe6529aa3db4312] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:25.916 INFO 2458 --- [alarm-detect-10,2d8c3cf42c335491,7e41f916837923b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:25.926 INFO 2458 --- [alarm-detect-10,2d8c3cf42c335491,7e41f916837923b0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:25.943 INFO 2458 --- [alarm-detect-12,2d8c3cf42c335491,458cae4e8364d280] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535080 ~ 1761535095 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535095, value=54.537)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535095, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535095=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:18:26.362 INFO 2458 --- [ocp-schedules-2,75995294a6e06352,8c76f323e373dec5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:26.550 WARN 2458 --- [ocp-async-3,14bbd06efc55be75,5b13c802342055fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:26.550 INFO 2458 --- [ocp-async-10,7249aa610306bc4c,578649c54184ce67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:26.707 INFO 2458 --- [prometheus-minute-schedule-1,17f63f9b6135ca95,4d429941d161d771] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:26.714 INFO 2458 --- [prometheus-minute-schedule-1,17f63f9b6135ca95,4d429941d161d771] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:27.091 INFO 2458 --- [ocp-schedules-18,89bbc5f51af70957,f2654db7d099bc16] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:18:27.102 INFO 2458 --- [ocp-schedules-18,89bbc5f51af70957,f2654db7d099bc16] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:27.102 INFO 2458 --- [ocp-schedules-18,89bbc5f51af70957,f2654db7d099bc16] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:27.224 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:27.225 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:28.519 INFO 2458 --- [ocp-async-10,5e2043ff162a8b2d,a7592fb5e7426bbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:28.550 WARN 2458 --- [ocp-async-8,16c7ac9b00a4e338,27bd5c5ce0354281] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:28.550 INFO 2458 --- [ocp-async-8,4e57867eb20a952d,cd45ce3d46bcd4ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:28.929 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:18:28.929 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:18:28.933 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:18:28.933 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:18:28.938 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:18:28.938 INFO 2458 --- [ocp-schedules-12,0ddd857884e9273f,dcce715082394a75] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:18:29.391 INFO 2458 --- [ocp-schedules-23,b8fa8b964bfbe875,ad79c3efc266a770] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:18:29.402 INFO 2458 --- [ocp-schedules-23,b8fa8b964bfbe875,ad79c3efc266a770] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:18:29.402 INFO 2458 --- [ocp-schedules-23,b8fa8b964bfbe875,ad79c3efc266a770] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:18:29.707 INFO 2458 --- [prometheus-minute-schedule-1,91afa2e6b2ec0a6a,30ebb6546137aec0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:29.714 INFO 2458 --- [prometheus-minute-schedule-1,91afa2e6b2ec0a6a,30ebb6546137aec0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:30.225 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:30.226 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:30.550 WARN 2458 --- [ocp-async-1,79e76c73b79e4970,eb032a3ecf4e8a36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:30.550 INFO 2458 --- [ocp-async-5,8500246459838dbc,ee4450d6149ba72b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:31.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:31.713171+08:00, lastProcessTime=2025-10-27T11:18:21.713170+08:00 2025-10-27 11:18:31.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:31.713171+08:00 2025-10-27 11:18:32.550 WARN 2458 --- [ocp-async-7,e4380e799e230a2a,3b1530c3abf29352] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:32.550 INFO 2458 --- [ocp-async-7,4de58e07a4674c4e,98f99f6611eb982a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:32.707 INFO 2458 --- [prometheus-minute-schedule-1,908dee2a36f30015,61e3a9ac1556c8ab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:32.715 INFO 2458 --- [prometheus-minute-schedule-1,908dee2a36f30015,61e3a9ac1556c8ab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:33.226 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:33.226 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:33.831 INFO 2458 --- [ocp-schedules-14,ca379e35c6611dc6,c86b534d80212372] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:18:33.831 INFO 2458 --- [ocp-schedules-14,ca379e35c6611dc6,c86b534d80212372] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:18:34.168 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:18:34.168 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:18:34.192 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:18:34.192 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:18:34.192 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:18:34.207 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:18:34.207 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:18:34.207 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:18:34.223 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:18:34.223 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:18:34.223 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:18:34.234 INFO 2458 --- [ocp-schedules-26,6f85499922ff8f38,35567f30052035bf] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:18:34.301 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6c516b263fa1053a,619d4998f5f3f02b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=6c516b263fa1053a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:18:34.301 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,6c516b263fa1053a,619d4998f5f3f02b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=6c516b263fa1053a, duration=1 ms] 2025-10-27 11:18:34.442 INFO 2458 --- [ocp-schedules-20,9702a17096b0e503,3bc3a548c0ace17e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:34.446 INFO 2458 --- [ocp-schedules-20,9702a17096b0e503,3bc3a548c0ace17e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:34.550 WARN 2458 --- [ocp-async-10,df77001c655dc659,646d64ba1e32b156] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:34.551 INFO 2458 --- [ocp-async-4,d18b1af53e566af8,8b7d825d4cf63ad6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:35.707 INFO 2458 --- [prometheus-minute-schedule-1,5d18d913be572cb2,b504741f20c14a78] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:35.715 INFO 2458 --- [prometheus-minute-schedule-1,5d18d913be572cb2,b504741f20c14a78] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:35.948 INFO 2458 --- [ocp-schedules-21,8c644a1fe1f1e289,e54bff91cfe8852a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:35.992 WARN 2458 --- [scanner-112166336-1,8c644a1fe1f1e289,06a09ec1a430a1ff] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:36.226 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:36.227 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:36.367 INFO 2458 --- [ocp-schedules-24,dcc3588551eba065,983229e03e69fd0f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:36.551 WARN 2458 --- [ocp-async-8,616b6f668ec5f45f,f8ced3aaeb2c5b6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:36.551 INFO 2458 --- [ocp-async-9,22c23d488fb7dfdd,054c54453e7b4c1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:37.183 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:18:37.183 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:18:37.284 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:37.285 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:18:37.298 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:37.298 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:18:37.361 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:37.362 INFO 2458 --- [ocp-schedules-26,45b7f0f72f8ffd3a,bb3d4f4bbd3d8a1a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:18:37.708 INFO 2458 --- [prometheus-cache-monitor1,9427c20294a4141d,579f20b251a5e7e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20394,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:18:37.708 INFO 2458 --- [prometheus-cache-monitor1,9427c20294a4141d,579f20b251a5e7e1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:18:38.303 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:38.303 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:18:38.303 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:18:38.304 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:18:38.304 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:18:38.304 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:18:38.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:38.318 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:38.318 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:38.319 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:38.320 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:38.322 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:38.323 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:38.323 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:38.323 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:38.324 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:38.377 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:18:38.378 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:18:38.378 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:18:38.380 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:18:38.388 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:18:38.390 INFO 2458 --- [ocp-schedules-19,2c744becfa3e3a95,6c4c53d0ab1349e4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:18:38.519 INFO 2458 --- [ocp-async-9,c92394710baec2f4,bb7b40599de5b388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:38.551 WARN 2458 --- [ocp-async-5,e553116ddae4671e,146740757c9df364] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:38.551 INFO 2458 --- [ocp-async-3,46e0fafe341ca27a,49cb01d97784a2c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:38.677 INFO 2458 --- [ocp-schedules-7,3c42c33195127f47,71482c98f2f38343] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:18:38.683 INFO 2458 --- [ocp-schedules-7,3c42c33195127f47,71482c98f2f38343] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:38.683 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:18:38.683 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:18:38.683 INFO 2458 --- [ocp-schedules-7,3c42c33195127f47,71482c98f2f38343] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:18:38.707 INFO 2458 --- [prometheus-minute-schedule-1,e585cf8f4fd51727,a51a4e74b5f1d6b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:38.714 INFO 2458 --- [prometheus-minute-schedule-1,e585cf8f4fd51727,a51a4e74b5f1d6b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:39.227 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:39.228 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:40.008 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:18:40.008 INFO 2458 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:18:40.051 INFO 2458 --- [Check obproxy1,d54edc490fcfd1a9,fe4570d96305b77c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:18:40.051 INFO 2458 --- [Check obproxy1,d54edc490fcfd1a9,fe4570d96305b77c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:18:40.053 INFO 2458 --- [Collect obproxy1,4701de0cad115aeb,1c8afd6819344425] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:18:40.053 INFO 2458 --- [Collect obproxy1,4701de0cad115aeb,1c8afd6819344425] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:18:40.083 INFO 2458 --- [Check obproxy1,d54edc490fcfd1a9,fe4570d96305b77c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:18:40.083 INFO 2458 --- [Collect obproxy1,4701de0cad115aeb,1c8afd6819344425] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:18:40.086 INFO 2458 --- [Check obproxy1,d54edc490fcfd1a9,fe4570d96305b77c] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:18:40.086 INFO 2458 --- [Check obproxy1,d54edc490fcfd1a9,fe4570d96305b77c] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:18:40.087 INFO 2458 --- [Collect obproxy1,4701de0cad115aeb,1c8afd6819344425] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:18:40.089 INFO 2458 --- [Collect obproxy1,4701de0cad115aeb,1c8afd6819344425] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:18:40.551 WARN 2458 --- [ocp-async-10,da35b7e6388caa5a,d87c9473d8689bd1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:40.551 INFO 2458 --- [ocp-async-6,9c58270d05f16476,a9470414493e7c2d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:40.995 INFO 2458 --- [ocp-schedules-17,7a7120ac2a5e4121,72a1e16ca7288c22] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:18:40.996 INFO 2458 --- [ocp-schedules-17,7a7120ac2a5e4121,72a1e16ca7288c22] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:18:41.707 INFO 2458 --- [prometheus-minute-schedule-1,388f6dc49c9f733d,e12ed540a5ff5230] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:41.714 INFO 2458 --- [prometheus-minute-schedule-1,388f6dc49c9f733d,e12ed540a5ff5230] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:41.714078+08:00, lastProcessTime=2025-10-27T11:18:31.713171+08:00 2025-10-27 11:18:41.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:41.714078+08:00 2025-10-27 11:18:42.228 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:42.229 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:42.551 WARN 2458 --- [ocp-async-8,3b321bafdeddcf76,4fc1089ba1f134d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:42.551 INFO 2458 --- [ocp-async-1,38ee094e398f3db8,592467b079f03ba3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:44.098 INFO 2458 --- [ocp-schedules-19,7b6714ee3f73719b,7c95de6ca3ada1df] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:18:44.454 INFO 2458 --- [ocp-schedules-24,31288be0559d43bc,5e83e63310b9bb6b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:44.458 INFO 2458 --- [ocp-schedules-24,31288be0559d43bc,5e83e63310b9bb6b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:44.551 WARN 2458 --- [ocp-async-1,f001297f3fedbf7b,01ab838c7874f88e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:44.551 INFO 2458 --- [ocp-async-1,ad67a6b087e97875,071139d60099d4d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:44.707 INFO 2458 --- [prometheus-minute-schedule-1,93167a97a4e4e237,06a8803ae0d92018] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:44.714 INFO 2458 --- [prometheus-minute-schedule-1,93167a97a4e4e237,06a8803ae0d92018] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:45.102 INFO 2458 --- [prometheus-second-schedule-1,aa6003b4676184f7,22dd428d1ee3f765] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:18:45.229 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:45.230 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:46.112 INFO 2458 --- [ocp-schedules-25,8e71f29f66f79495,0df26f570711b01b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:46.260 WARN 2458 --- [scanner-112166516-1,8e71f29f66f79495,650a29febc796091] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:46.312 INFO 2458 --- [alarm-detect-6,8e71f29f66f79495,fac026e9581fbb64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:46.316 INFO 2458 --- [alarm-detect-6,8e71f29f66f79495,fac026e9581fbb64] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:18:46.337 INFO 2458 --- [alarm-detect-9,8e71f29f66f79495,294218cf14049820] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:46.340 INFO 2458 --- [alarm-detect-6,8e71f29f66f79495,a9f6aa05ff97c4e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:46.341 INFO 2458 --- [alarm-detect-9,8e71f29f66f79495,294218cf14049820] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:18:46.344 INFO 2458 --- [alarm-detect-6,8e71f29f66f79495,a9f6aa05ff97c4e0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:18:46.381 INFO 2458 --- [ocp-schedules-26,547ad9b35aad6514,ea6189fa02f13095] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:46.395 INFO 2458 --- [alarm-detect-17,8e71f29f66f79495,bb4a0cef2be2d057] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535056 ~ 1761535116 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535080, value=203414.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203414.56, values={1761535080=203414.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:18:46.551 WARN 2458 --- [ocp-async-1,06ad0cfd7885b109,b85878ebab6cc2a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:46.551 INFO 2458 --- [ocp-async-4,9edef6743b4588c8,0d53fb76915558ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:46.657 INFO 2458 --- [alarm-detect-15,8e71f29f66f79495,f919ecdbda65cd37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:46.661 INFO 2458 --- [alarm-detect-15,8e71f29f66f79495,f919ecdbda65cd37] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:18:46.664 INFO 2458 --- [alarm-detect-17,8e71f29f66f79495,d10ad9a01e69d4ef] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535056 ~ 1761535116 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535080, value=93.072)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.072, values={1761535080=93.072}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:18:46.697 INFO 2458 --- [alarm-detect-11,8e71f29f66f79495,917c72da648c5e5d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535056 ~ 1761535116 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535080, value=203415.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203415.912, values={1761535080=203415.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:18:46.806 INFO 2458 --- [alarm-detect-19,8e71f29f66f79495,e02b1e83f20251d9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535095 ~ 1761535116 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535110, value=45.991)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535110, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535110=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:18:47.106 INFO 2458 --- [ocp-schedules-28,4cb067c3e0f811d7,9513c9fd157afce3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:18:47.115 INFO 2458 --- [ocp-schedules-28,4cb067c3e0f811d7,9513c9fd157afce3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:47.115 INFO 2458 --- [ocp-schedules-28,4cb067c3e0f811d7,9513c9fd157afce3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:18:47.707 INFO 2458 --- [prometheus-minute-schedule-1,10896b364e5b6d6d,c6acbc48a03b9958] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:47.714 INFO 2458 --- [prometheus-minute-schedule-1,10896b364e5b6d6d,c6acbc48a03b9958] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:48.230 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:48.231 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:48.520 INFO 2458 --- [ocp-async-4,e3e39f5980e82a43,ebb4106d71a55c4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:48.552 WARN 2458 --- [ocp-async-9,75b616bee7d3f4b1,bbfb138e6edf1802] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:48.552 INFO 2458 --- [ocp-async-6,7e43aee800b78b4d,fe98b2c0e393474e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:48.679 INFO 2458 --- [ocp-schedules-23,72fef9817c37e7f1,ab3c0b91cbdebae8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:18:48.679 INFO 2458 --- [ocp-schedules-23,72fef9817c37e7f1,ab3c0b91cbdebae8] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:18:48.688 INFO 2458 --- [ocp-schedules-23,72fef9817c37e7f1,ab3c0b91cbdebae8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:18:48.688 INFO 2458 --- [ocp-schedules-23,72fef9817c37e7f1,ab3c0b91cbdebae8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:18:48.695 INFO 2458 --- [ocp-schedules-1,7462d676aa3927e4,c24ade73118776ae] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:18:48.695+08:00, clusters=[1, 2] 2025-10-27 11:18:48.726 INFO 2458 --- [ocp-schedules-1,7462d676aa3927e4,c24ade73118776ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:18:48.729 INFO 2458 --- [ocp-schedules-1,7462d676aa3927e4,c24ade73118776ae] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:18:50.552 WARN 2458 --- [ocp-async-2,e811ff467266be72,7d7eac7a97066765] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:50.552 INFO 2458 --- [ocp-async-5,ff871c0caf2bd4ee,382516e65c1be7be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:50.707 INFO 2458 --- [prometheus-minute-schedule-1,7c94cb7bc81a54bf,75f039895a61b640] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:50.714 INFO 2458 --- [prometheus-minute-schedule-1,7c94cb7bc81a54bf,75f039895a61b640] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:51.231 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:51.232 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:51.408 INFO 2458 --- [ocp-schedules-19,23a5d94376409a6a,fbe4797535ce4b7a] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:51.408 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:18:51.408 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:18:51.408 INFO 2458 --- [ocp-schedules-10,665fdeb54a4ba76e,576bdbf061104b98] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:51.408 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:18:51.408 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:18:51.409 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:18:51.425 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:51.426 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:51.426 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:51.426 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:51.427 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:51.427 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:51.429 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:51.429 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:51.429 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:51.430 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:51.431 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:51.431 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:51.431 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:51.433 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:51.433 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:51.433 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:51.433 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:51.434 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:51.434 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:51.438 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:51.439 INFO 2458 --- [ocp-schedules-1,c0668e37b9848477,8d621b3c505519d2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:18:51.440 INFO 2458 --- [host-check-clockdiff1,c0668e37b9848477,ed133453ffb78033] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:18:51.440 INFO 2458 --- [host-check-clockdiff2,c0668e37b9848477,298cd69af7a89f4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:18:51.441 INFO 2458 --- [host-check-clockdiff3,c0668e37b9848477,6ab822078db6c499] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:18:51.441 INFO 2458 --- [host-check-clockdiff4,c0668e37b9848477,a5b2ea45acff84e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:18:51.441 INFO 2458 --- [host-check-clockdiff5,c0668e37b9848477,c962245f2b4bede9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:18:51.442 INFO 2458 --- [ocp-schedules-21,1113232c75765bb2,8bf9badb9ae6c7bb] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:18:51.449 INFO 2458 --- [host-check-clockdiff1,c0668e37b9848477,ed133453ffb78033] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535131 0 0 , error= 2025-10-27 11:18:51.456 INFO 2458 --- [host-check-clockdiff4,c0668e37b9848477,a5b2ea45acff84e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535131 0 0 , error= 2025-10-27 11:18:51.459 INFO 2458 --- [host-check-clockdiff2,c0668e37b9848477,298cd69af7a89f4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535131 -1 -1 , error= 2025-10-27 11:18:51.481 INFO 2458 --- [host-check-clockdiff3,c0668e37b9848477,6ab822078db6c499] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535131 -1 -1 , error= 2025-10-27 11:18:51.510 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:18:51.511 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:18:51.512 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:18:51.512471236+08:00 2025-10-27 11:18:51.513 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:18:51.513 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:18:51.513 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:18:51.512+08:00 2025-10-27 11:18:51.515 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:18:51.515663272+08:00 2025-10-27 11:18:51.516 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:18:51.516 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:18:51.515+08:00 2025-10-27 11:18:51.516 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:18:51.517 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:18:51.517 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:18:51.517 INFO 2458 --- [host-check-agent-status3,665fdeb54a4ba76e,d82e4f9fb9b5ac14] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:18:51.518 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:18:51.518 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:18:51.517734945+08:00 2025-10-27 11:18:51.519 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:18:51.518470326+08:00 2025-10-27 11:18:51.519 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:18:51.518+08:00 2025-10-27 11:18:51.520 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:18:51.520 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:18:51.520 INFO 2458 --- [host-check-agent-status4,665fdeb54a4ba76e,e09a3bd1401c7d75] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:18:51.520 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:18:51.519+08:00 2025-10-27 11:18:51.523 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:18:51.523 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:18:51.523 INFO 2458 --- [host-check-agent-status1,665fdeb54a4ba76e,cd74a68219f23e46] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:18:51.525 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:18:51.527 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:18:51.527 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:18:51.527 INFO 2458 --- [host-check-agent-status5,665fdeb54a4ba76e,d1e4b6cb22eb22fe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:18:51.528 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:18:51.512+08:00 and status AVAILABLE 2025-10-27 11:18:51.528 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:18:51.515+08:00 and status AVAILABLE 2025-10-27 11:18:51.530 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:18:51.518+08:00 and status AVAILABLE 2025-10-27 11:18:51.531 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:18:51.519+08:00 and status AVAILABLE 2025-10-27 11:18:51.533 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:18:51.535 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:18:51.535 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:18:51.536 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:18:51.539 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:18:51.539 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:18:51.539 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:18:51.538635998+08:00 2025-10-27 11:18:51.540 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:18:51.540+08:00 2025-10-27 11:18:51.541 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:18:51.541 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:18:51.541 INFO 2458 --- [host-check-agent-status2,665fdeb54a4ba76e,8418a7d4215136ae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:18:51.545 INFO 2458 --- [ocp-schedules-5,e3229d8469195771,2dcb0c20152a2187] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:18:51.549 INFO 2458 --- [ocp-schedules-11,e2d4150a488743d3,325278367afdb962] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:18:51.549 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:18:51.540+08:00 and status AVAILABLE 2025-10-27 11:18:51.549 INFO 2458 --- [ocp-schedules-5,e3229d8469195771,2dcb0c20152a2187] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:51.553 INFO 2458 --- [ocp-schedules-11,e2d4150a488743d3,325278367afdb962] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:51.553 INFO 2458 --- [ocp-schedules-11,e2d4150a488743d3,325278367afdb962] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:18:51.554 INFO 2458 --- [ocp-schedules-22,f5a9c25753feb4c2,142714a269f45f0b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:18:51.554 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:18:51.555 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:51.556 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:51.558 INFO 2458 --- [ocp-schedules-22,f5a9c25753feb4c2,142714a269f45f0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:18:51.558 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:51.559 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:51.560 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:18:51.562 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:18:51.565 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:18:51.565 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:18:51.568 INFO 2458 --- [ocp-schedules-6,268279cc409eec69,6a94345e07bf83cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:18:51.569 INFO 2458 --- [ocp-schedules-17,71d4feee72b6ccb9,7f922cc0ad5fa8fc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:18:51.578 INFO 2458 --- [ocp-schedules-4,7e6b4f17e725b897,27cdbfbc9c61ccda] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:18:51.580 INFO 2458 --- [ocp-schedules-6,268279cc409eec69,6a94345e07bf83cf] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:18:51.581 INFO 2458 --- [ocp-schedules-17,71d4feee72b6ccb9,7f922cc0ad5fa8fc] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:18:51.586 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:18:51.589 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:18:51.592 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:51.594 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:18:51.627 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.631 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:51.632 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:51.637 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:18:51.637 INFO 2458 --- [ocp-schedules-5,e3229d8469195771,2dcb0c20152a2187] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:18:51.637 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:51.643 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:18:51.650 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.650 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:51.651 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:18:51.658 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:18:51.660 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:18:51.662 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:18:51.680 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:18:51.684 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:18:51.691 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:18:51.691 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:18:51.693 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:18:51.708 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:51.708 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.709 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:51.709 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.709 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:18:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:18:51.715748+08:00, lastProcessTime=2025-10-27T11:18:41.714078+08:00 2025-10-27 11:18:51.725 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:18:51.731 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:51.731 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:51.738 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:51.746 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:18:51.751 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:18:51.753 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:51.763 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:18:51.773 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:18:51.773 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:18:51.776 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:18:51.778 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:18:51.789 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:18:51.799 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:51.800 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.800 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:51.800 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:51.801 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.801 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:18:51.801 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.804 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:18:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:51.807 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:18:51.817 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:18:51.820 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:51.820 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:51.820 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:18:51.825 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:51.825 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:51.835 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:18:51.840 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:18:51.842 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:51.849 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:51.850 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:18:51.851 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:51.866 INFO 2458 --- [host-check-host-status4,23a5d94376409a6a,f8f343530d946b5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:18:51.866 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:18:51.870 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:18:51.872 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:18:51.876 INFO 2458 --- [host-check-host-status3,23a5d94376409a6a,a71fd6c10c9d0bee] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:18:51.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:51.878 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.879 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:51.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:18:51.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:51.909 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:18:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:18:51.910 INFO 2458 --- [host-check-host-status5,23a5d94376409a6a,89709643e2477728] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:18:51.910 INFO 2458 --- [host-check-host-status2,23a5d94376409a6a,5b73cc63a9f59601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:18:51.911 INFO 2458 --- [host-check-host-status1,23a5d94376409a6a,26292b33d0804fe1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:18:51.912 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:18:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:51.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:18:51.922 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:18:51.954 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:51.954 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:51.973 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:18:51.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:51.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:51.997 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:18:51.715748+08:00 2025-10-27 11:18:52.017 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:18:52.018 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.019 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:18:52.020 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.021 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:18:52.028 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:52.028 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:52.047 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:18:52.084 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:52.084 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.085 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:18:52.103 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:52.103 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:18:52.118 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:18:52.118 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:18:52.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:52.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:18:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:18:52.229 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:52.230 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.231 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:18:52.235 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:18:52.244 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:18:52.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:18:52.256 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:18:52.303 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:18:52.373 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:18:52.374 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.374 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:18:52.376 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.377 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:18:52.421 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:18:52.464 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:18:52.464 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:18:52.465 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:18:52.503 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:18:52.503 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:18:52.503 INFO 2458 --- [ocp-schedules-22,f5a9c25753feb4c2,142714a269f45f0b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:18:52.552 INFO 2458 --- [ocp-async-1,1de53756740fc0f3,7c115e8d6f8d3da5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:52.552 WARN 2458 --- [ocp-async-4,392c8648e6b1d28a,93cd38e0f25f9169] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:18:53.234 INFO 2458 --- [host-check-clockdiff5,c0668e37b9848477,c962245f2b4bede9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535133 0 0 , error= 2025-10-27 11:18:53.707 INFO 2458 --- [prometheus-minute-schedule-1,e82c94ec71dfdc2f,85ee817cbc5bae7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:53.714 INFO 2458 --- [prometheus-minute-schedule-1,e82c94ec71dfdc2f,85ee817cbc5bae7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:54.232 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:54.232 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:54.466 INFO 2458 --- [ocp-schedules-26,5a1a1be95055ce56,f1034441c3bc8fba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:18:54.470 INFO 2458 --- [ocp-schedules-26,5a1a1be95055ce56,f1034441c3bc8fba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:18:54.552 INFO 2458 --- [ocp-async-7,96af5a9438762842,144ab853fede2b45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:54.552 WARN 2458 --- [ocp-async-2,1f7608a0de08734c,4c1acfafd35aa30c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:54.590 INFO 2458 --- [ocp-schedules-1,cdeeb5e866bb1508,3226afda892e9a18] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 1 2025-10-27 11:18:54.593 INFO 2458 --- [ocp-schedules-1,cdeeb5e866bb1508,3226afda892e9a18] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-10-27 11:18:54.593 INFO 2458 --- [ocp-schedules-1,cdeeb5e866bb1508,3226afda892e9a18] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 1 2025-10-27 11:18:54.932 INFO 2458 --- [ocp-schedules-27,4e49cbb972132c6c,e71ab790b9c6608d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:18:56.387 INFO 2458 --- [ocp-schedules-6,b661c7c99ada3ee2,f0e19801085561c5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:18:56.552 INFO 2458 --- [ocp-async-6,f1ccf163f295061b,6d4010af6c8f12d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:56.552 WARN 2458 --- [ocp-async-5,44a5f30a08c37b4b,a392a6e15bd0e133] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:56.707 INFO 2458 --- [prometheus-minute-schedule-1,dab60f92941df841,f4348d61a5f240d7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:56.713 INFO 2458 --- [prometheus-minute-schedule-1,dab60f92941df841,f4348d61a5f240d7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:18:56.813 INFO 2458 --- [ocp-schedules-22,b348ee5363af252a,27a3d6f141a22bb2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:18:56.881 WARN 2458 --- [scanner-112166802-1,b348ee5363af252a,e5046b555b03ae5e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:18:57.089 INFO 2458 --- [alarm-detect-3,b348ee5363af252a,0393ebeda6be8ec5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535110 ~ 1761535126 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535125, value=41.972)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535125, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535125=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:18:57.233 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:18:57.233 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:18:58.520 INFO 2458 --- [ocp-async-3,d6d90865b137f0df,f38163e1a4622a51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:18:58.552 INFO 2458 --- [ocp-async-4,5f5fa1980d837020,2e5874cc83b10b84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:18:58.552 WARN 2458 --- [ocp-async-9,30c5d6068eddf1f9,060a8ebf42f4a102] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:18:59.339 INFO 2458 --- [ocp-schedules-12,42d1927df5c8c3e7,9aa9f2d52c01d18e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:18:59.595 INFO 2458 --- [ocp-schedules-1,0b55dd5777eb13d4,180184d2f3e6e94c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:18:59.600 INFO 2458 --- [ocp-schedules-1,0b55dd5777eb13d4,180184d2f3e6e94c] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:18:59.707 INFO 2458 --- [prometheus-minute-schedule-1,1902a895221b3c2e,296d8adb68a54f9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535140, interval=60 2025-10-27 11:18:59.714 INFO 2458 --- [prometheus-minute-schedule-1,1902a895221b3c2e,296d8adb68a54f9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:00.233 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:00.234 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:00.552 INFO 2458 --- [ocp-async-2,af9100235738d541,1479c51797d16f5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:00.552 WARN 2458 --- [ocp-async-8,66ffca38da944e01,85117600dc67805e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:01.713630+08:00, lastProcessTime=2025-10-27T11:18:51.715748+08:00 2025-10-27 11:19:01.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:01.713630+08:00 2025-10-27 11:19:02.552 WARN 2458 --- [ocp-async-5,3e3be3fc32d41e2f,7f61c32bba691496] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:02.553 INFO 2458 --- [ocp-async-10,bf5ab63001f21458,489be68dd7444e53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:02.707 INFO 2458 --- [prometheus-minute-schedule-1,0401328ae8f89296,2abffbe0088464d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:02.716 INFO 2458 --- [prometheus-minute-schedule-1,0401328ae8f89296,2abffbe0088464d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:03.234 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:03.235 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:03.363 INFO 2458 --- [ocp-schedules-6,e005e17f58c320cd,1b0756180322abf9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:19:03.371 INFO 2458 --- [ocp-schedules-6,e005e17f58c320cd,1b0756180322abf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:03.422 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:03.423 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:19:03.432 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:03.432 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:19:03.446 INFO 2458 --- [ocp-schedules-6,e005e17f58c320cd,1b0756180322abf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:19:04.482 INFO 2458 --- [ocp-schedules-25,67c7f1d14207887a,daa32ebeeb8e8a50] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:04.486 INFO 2458 --- [ocp-schedules-25,67c7f1d14207887a,daa32ebeeb8e8a50] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:04.553 WARN 2458 --- [ocp-async-3,e0ebc6d42d2828b1,29947a59734dae61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:04.553 INFO 2458 --- [ocp-async-4,1a733a022dad23fa,cb22d55975a30f91] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:05.074 INFO 2458 --- [ocp-schedules-17,f7d6a74325c599aa,346cf3bf27a30a4b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:19:05.082 INFO 2458 --- [ocp-schedules-17,f7d6a74325c599aa,346cf3bf27a30a4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:05.173 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:19:05.173 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.176 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:19:05.185 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.185 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.211 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.212 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.238 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.238 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.240 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.241 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.247 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:19:05.247 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:19:05.265 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.265 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.292 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.292 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.293 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.293 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.320 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.320 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.321 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:19:05.321 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.322 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.323 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.323 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.323 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.324 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.324 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.325 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.326 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.326 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.327 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.328 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.329 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.329 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:19:05.332 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:19:05.353 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.354 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.377 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:05.378 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.378 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:19:05.381 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.381 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:19:05.383 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.384 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:19:05.388 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.388 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:19:05.388 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.388 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:19:05.391 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:19:05.391 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:19:05.417 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.418 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:19:05.446 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.447 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:19:05.449 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.449 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:19:05.476 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.476 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:19:05.504 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.504 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:19:05.551 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.552 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.553 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.554 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.554 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.555 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.556 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.557 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.557 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.558 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.559 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.560 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.560 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.561 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.562 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.563 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.563 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.564 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.565 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.565 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:19:05.566 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.567 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:19:05.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.569 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:19:05.569 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:19:05.572 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:19:05.666 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:05.667 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:05.668 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:19:05.692 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:19:05.695 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:19:05.695 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:19:05.695 INFO 2458 --- [ocp-schedules-17,f7d6a74325c599aa,346cf3bf27a30a4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:19:05.707 INFO 2458 --- [prometheus-minute-schedule-1,2ed30b6d61301bed,e0721e983ccc8013] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:05.714 INFO 2458 --- [prometheus-minute-schedule-1,2ed30b6d61301bed,e0721e983ccc8013] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:06.235 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:06.236 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:06.393 INFO 2458 --- [ocp-schedules-3,e117cad30caad80c,6fee4955683d9000] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:06.553 WARN 2458 --- [ocp-async-8,5bc6ce4259c3c9b0,45809c9161e4c9e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:06.553 INFO 2458 --- [ocp-async-7,2ce34f9424a9ea3c,4b2fbc14c4472d96] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:07.096 INFO 2458 --- [ocp-schedules-12,316ad11ac40fc54c,b364dd6fbecd1bc5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:07.117 INFO 2458 --- [ocp-schedules-24,0261a8f42132c37a,59864c0c917aeaf6] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:19:07.125 INFO 2458 --- [ocp-schedules-24,0261a8f42132c37a,59864c0c917aeaf6] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:07.125 INFO 2458 --- [ocp-schedules-24,0261a8f42132c37a,59864c0c917aeaf6] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:07.131 WARN 2458 --- [scanner-112166974-1,316ad11ac40fc54c,b6a8abcb95a5dcca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:07.156 INFO 2458 --- [alarm-detect-14,316ad11ac40fc54c,a6ff83720337e27d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:07.160 INFO 2458 --- [alarm-detect-14,316ad11ac40fc54c,a6ff83720337e27d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:07.167 INFO 2458 --- [alarm-detect-9,316ad11ac40fc54c,aba105666058b180] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:07.171 INFO 2458 --- [alarm-detect-9,316ad11ac40fc54c,aba105666058b180] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:07.172 INFO 2458 --- [alarm-detect-8,316ad11ac40fc54c,12924dc097651d11] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:07.176 INFO 2458 --- [alarm-detect-8,316ad11ac40fc54c,12924dc097651d11] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:07.263 INFO 2458 --- [alarm-detect-13,316ad11ac40fc54c,10e2b015ca7f8cff] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:07.267 INFO 2458 --- [alarm-detect-13,316ad11ac40fc54c,10e2b015ca7f8cff] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:08.520 INFO 2458 --- [ocp-async-6,d316ec2cc464e212,655794a24e325200] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:08.553 WARN 2458 --- [ocp-async-5,8a60f69934237c7d,152f53126f2cde1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:08.553 INFO 2458 --- [ocp-async-9,ae09526753697027,777506caac3ef012] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:08.707 INFO 2458 --- [prometheus-minute-schedule-1,f324af7b04ed93a8,c44ff60de385b71e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:08.714 INFO 2458 --- [prometheus-minute-schedule-1,f324af7b04ed93a8,c44ff60de385b71e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:09.236 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:09.237 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:10.014 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:19:10.014 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:19:10.062 INFO 2458 --- [Check obproxy1,f7e91e028c8159e5,bd1f957f0e825a8f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:19:10.062 INFO 2458 --- [Check obproxy1,f7e91e028c8159e5,bd1f957f0e825a8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:19:10.063 INFO 2458 --- [Collect obproxy1,2f62c6843107bca9,cc10ea1f183570a5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:19:10.063 INFO 2458 --- [Collect obproxy1,2f62c6843107bca9,cc10ea1f183570a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:19:10.095 INFO 2458 --- [Collect obproxy1,2f62c6843107bca9,cc10ea1f183570a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:19:10.098 INFO 2458 --- [Check obproxy1,f7e91e028c8159e5,bd1f957f0e825a8f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:19:10.098 INFO 2458 --- [Collect obproxy1,2f62c6843107bca9,cc10ea1f183570a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:19:10.102 INFO 2458 --- [Check obproxy1,f7e91e028c8159e5,bd1f957f0e825a8f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:19:10.102 INFO 2458 --- [Check obproxy1,f7e91e028c8159e5,bd1f957f0e825a8f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:19:10.102 INFO 2458 --- [Collect obproxy1,2f62c6843107bca9,cc10ea1f183570a5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:19:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:19:10.553 WARN 2458 --- [ocp-async-1,233c76ad37df1f93,c801f2e7022dabfa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:10.553 INFO 2458 --- [ocp-async-4,f877ec83ce56f3fc,041106256bc79b02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:11.070 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535080, nodeCount=821, affectRows=821 2025-10-27 11:19:11.237 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=1000, affectRows=1000 2025-10-27 11:19:11.241 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=994, validCount=992 2025-10-27 11:19:11.293 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=1000, affectRows=1000 2025-10-27 11:19:11.298 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=999, validCount=995 2025-10-27 11:19:11.313 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134521640) Duplicate entry '835014129065861120-1761535020' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134521640) Duplicate entry '835014129065861120-1761535020' for key 'PRIMARY' 2025-10-27 11:19:11.707 INFO 2458 --- [prometheus-minute-schedule-1,7fc34e596f3224be,bd03cdb59324fdd5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:11.714 INFO 2458 --- [prometheus-minute-schedule-1,7fc34e596f3224be,bd03cdb59324fdd5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:11.713217+08:00, lastProcessTime=2025-10-27T11:19:01.713630+08:00 2025-10-27 11:19:11.925 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:11.713217+08:00 2025-10-27 11:19:12.237 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:12.237 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:12.553 INFO 2458 --- [ocp-async-8,803441adcbdd0613,9a8960193cdb97c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:12.553 WARN 2458 --- [ocp-async-7,2935cfb324261f42,ccbd71733c058f45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:12.797 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=1000, affectRows=1000 2025-10-27 11:19:12.801 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=1000, validCount=996 2025-10-27 11:19:12.806 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=44, affectRows=44 2025-10-27 11:19:12.807 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=43, validCount=43 2025-10-27 11:19:12.809 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=55, affectRows=55 2025-10-27 11:19:12.810 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=54, validCount=54 2025-10-27 11:19:12.823 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=1000, affectRows=1000 2025-10-27 11:19:12.827 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=996, validCount=992 2025-10-27 11:19:14.493 INFO 2458 --- [ocp-schedules-27,cd98023dc90f80dc,f5628d0eeb42d2fa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:14.497 INFO 2458 --- [ocp-schedules-27,cd98023dc90f80dc,f5628d0eeb42d2fa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:14.553 WARN 2458 --- [ocp-async-10,b0046e508d3fd848,4149a9144f5619e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:14.553 INFO 2458 --- [ocp-async-5,b8fa311df168778d,f604a1bc1603e53f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:14.707 INFO 2458 --- [prometheus-minute-schedule-1,71caba2a184fb904,48252a8a3b43b720] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:14.714 INFO 2458 --- [prometheus-minute-schedule-1,71caba2a184fb904,48252a8a3b43b720] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:15.107 INFO 2458 --- [ocp-schedules-26,0e983f195455f361,ae4ea8281481240f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:19:15.107+08:00, clusters=[1, 2] 2025-10-27 11:19:15.114 INFO 2458 --- [ocp-schedules-26,0e983f195455f361,ae4ea8281481240f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:19:15.119 INFO 2458 --- [ocp-schedules-26,0e983f195455f361,ae4ea8281481240f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:19:15.238 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:15.238 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:15.542 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535080, nodeCount=1000, affectRows=1000 2025-10-27 11:19:15.546 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535080, nodeCount=996, validCount=984 2025-10-27 11:19:15.742 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,ee913e92cf9fd81b,cbfa48e2f99222fa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=ee913e92cf9fd81b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:19:15.742 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,ee913e92cf9fd81b,cbfa48e2f99222fa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=ee913e92cf9fd81b, duration=1 ms] 2025-10-27 11:19:16.053 INFO 2458 --- [ocp-schedules-14,ba87a5d8981c3805,be4c56bcf99603cb] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:19:16.053 INFO 2458 --- [ocp-schedules-14,ba87a5d8981c3805,be4c56bcf99603cb] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:19:16.125 INFO 2458 --- [Collect all tenants major compactions1,ba87a5d8981c3805,691a4ee9ecb70bd9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:19:16.126 INFO 2458 --- [Collect all tenants major compactions2,ba87a5d8981c3805,742d16693c637aea] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:19:16.127 INFO 2458 --- [Collect all tenants major compactions3,ba87a5d8981c3805,a2fd6c3e30af4ca0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:19:16.128 INFO 2458 --- [Collect all tenants major compactions4,ba87a5d8981c3805,8a3f9732f4a4f5b5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:19:16.128 INFO 2458 --- [Collect all tenants major compactions6,ba87a5d8981c3805,4a079f2184fb2bc4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:19:16.130 INFO 2458 --- [Collect all tenants major compactions5,ba87a5d8981c3805,3bbaf3b6feb9245b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:19:16.179 INFO 2458 --- [Collect all tenants major compactions5,ba87a5d8981c3805,3bbaf3b6feb9245b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.181 INFO 2458 --- [Collect all tenants major compactions5,ba87a5d8981c3805,3bbaf3b6feb9245b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:19:16.186 INFO 2458 --- [Collect all tenants major compactions5,ba87a5d8981c3805,3bbaf3b6feb9245b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:19:16.187 INFO 2458 --- [Collect all tenants major compactions6,ba87a5d8981c3805,4a079f2184fb2bc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.188 INFO 2458 --- [Collect all tenants major compactions6,ba87a5d8981c3805,4a079f2184fb2bc4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:19:16.191 INFO 2458 --- [Collect all tenants major compactions6,ba87a5d8981c3805,4a079f2184fb2bc4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:19:16.200 INFO 2458 --- [Collect all tenants major compactions4,ba87a5d8981c3805,8a3f9732f4a4f5b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.202 INFO 2458 --- [Collect all tenants major compactions4,ba87a5d8981c3805,8a3f9732f4a4f5b5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:19:16.203 INFO 2458 --- [Collect all tenants major compactions1,ba87a5d8981c3805,691a4ee9ecb70bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.205 INFO 2458 --- [Collect all tenants major compactions1,ba87a5d8981c3805,691a4ee9ecb70bd9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:19:16.207 INFO 2458 --- [Collect all tenants major compactions4,ba87a5d8981c3805,8a3f9732f4a4f5b5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:19:16.209 INFO 2458 --- [Collect all tenants major compactions1,ba87a5d8981c3805,691a4ee9ecb70bd9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:19:16.210 INFO 2458 --- [Collect all tenants major compactions2,ba87a5d8981c3805,742d16693c637aea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.211 INFO 2458 --- [Collect all tenants major compactions3,ba87a5d8981c3805,a2fd6c3e30af4ca0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:16.212 INFO 2458 --- [Collect all tenants major compactions2,ba87a5d8981c3805,742d16693c637aea] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:19:16.213 INFO 2458 --- [Collect all tenants major compactions3,ba87a5d8981c3805,a2fd6c3e30af4ca0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:19:16.217 INFO 2458 --- [Collect all tenants major compactions2,ba87a5d8981c3805,742d16693c637aea] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:19:16.219 INFO 2458 --- [Collect all tenants major compactions3,ba87a5d8981c3805,a2fd6c3e30af4ca0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:19:16.220 INFO 2458 --- [ocp-schedules-14,ba87a5d8981c3805,be4c56bcf99603cb] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:19:16.401 INFO 2458 --- [ocp-schedules-25,995c763ba84d828e,dca470399751a2fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:16.553 INFO 2458 --- [ocp-async-9,c6114a1fe00d6ea5,a8ce70a2aebeea67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:16.554 WARN 2458 --- [ocp-async-3,cd6f09b336197b12,447db0827e06f380] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:17.315 INFO 2458 --- [ocp-schedules-22,28963f5706a320bb,56f4cd77630e4d84] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:17.376 WARN 2458 --- [scanner-112168769-1,28963f5706a320bb,abe2448f68ce3bb1] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:17.462 INFO 2458 --- [alarm-detect-6,28963f5706a320bb,21b1dfaf8e3a5f90] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535125 ~ 1761535147 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535140, value=49.262)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535140, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535140=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:19:17.707 INFO 2458 --- [prometheus-minute-schedule-1,78ce9ae1ab54d137,5f886635d7353286] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:17.714 INFO 2458 --- [prometheus-minute-schedule-1,78ce9ae1ab54d137,5f886635d7353286] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:18.238 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:18.239 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:18.520 INFO 2458 --- [ocp-async-9,365d1bc3e904b18f,62663c0e187c01f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:18.554 WARN 2458 --- [ocp-async-2,1c7a9fc4518cd917,b34476ddc5013a5a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:18.554 INFO 2458 --- [ocp-async-7,9d28ba53a671c908,8dbab395d67229c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:18.833 INFO 2458 --- [ocp-schedules-14,59dee6686664d6a4,560f67c524e433ae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:19:18.833 INFO 2458 --- [ocp-schedules-14,59dee6686664d6a4,560f67c524e433ae] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:19:18.841 INFO 2458 --- [ocp-schedules-14,59dee6686664d6a4,560f67c524e433ae] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:19:18.841 INFO 2458 --- [ocp-schedules-14,59dee6686664d6a4,560f67c524e433ae] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:19:19.860 INFO 2458 --- [ocp-schedules-9,e9eb05ce0261911e,08302c83e9db5ae4] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:19:20.554 INFO 2458 --- [ocp-async-10,58cb749028e897ea,2f0b10964a216842] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:20.554 WARN 2458 --- [ocp-async-5,3a256963793339ea,3b3b7ef217178375] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:20.707 INFO 2458 --- [prometheus-minute-schedule-1,04f92d3f7e4d5a9d,0ec365a4b65e0b54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:20.715 INFO 2458 --- [prometheus-minute-schedule-1,04f92d3f7e4d5a9d,0ec365a4b65e0b54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:21.239 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:21.240 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:21.408 INFO 2458 --- [ocp-schedules-11,6b425286608b6dfe,9cf942785cfe2a57] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:21.408 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:19:21.408 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:19:21.408 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:19:21.408 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [ocp-schedules-3,ba5b112be8653986,5c50b1a734d5b6fa] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:19:21.409 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:19:21.425 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:21.425 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:21.425 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:21.425 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:21.426 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:21.426 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:21.427 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:21.428 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:21.431 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:21.431 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:21.431 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:21.431 INFO 2458 --- [ocp-schedules-10,01d55f0c49424100,96ddf40db8af958b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:21.432 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:21.432 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:21.432 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:21.432 INFO 2458 --- [host-check-clockdiff1,01d55f0c49424100,878a33d9636d408b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:19:21.432 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:21.433 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:21.433 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:21.435 INFO 2458 --- [host-check-clockdiff2,01d55f0c49424100,bd3bd9ed6bd54afd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:19:21.435 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:21.436 INFO 2458 --- [host-check-clockdiff3,01d55f0c49424100,43fdb78adb1849e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:19:21.444 INFO 2458 --- [host-check-clockdiff4,01d55f0c49424100,e81c2df0face7a8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:19:21.444 INFO 2458 --- [host-check-clockdiff5,01d55f0c49424100,cabef0e5971a1074] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:19:21.444 INFO 2458 --- [ocp-schedules-8,5209d848ea6a84f5,6c83ed3d1c3a5542] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:19:21.445 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:21.446 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:21.453 INFO 2458 --- [host-check-clockdiff4,01d55f0c49424100,e81c2df0face7a8b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535161 -1 -1 , error= 2025-10-27 11:19:21.468 INFO 2458 --- [host-check-clockdiff2,01d55f0c49424100,bd3bd9ed6bd54afd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535161 -1 -1 , error= 2025-10-27 11:19:21.468 INFO 2458 --- [host-check-clockdiff1,01d55f0c49424100,878a33d9636d408b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535161 0 0 , error= 2025-10-27 11:19:21.468 INFO 2458 --- [host-check-clockdiff3,01d55f0c49424100,43fdb78adb1849e2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535161 -1 -1 , error= 2025-10-27 11:19:21.486 INFO 2458 --- [host-check-clockdiff5,01d55f0c49424100,cabef0e5971a1074] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535161 0 0 , error= 2025-10-27 11:19:21.504 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:19:21.509 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:19:21.509 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:19:21.509 INFO 2458 --- [host-check-agent-status2,ba5b112be8653986,23394e2992c0637a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:19:21.512 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:19:21.512 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:19:21.514 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:19:21.513998514+08:00 2025-10-27 11:19:21.514 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:19:21.514 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:19:21.515 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:19:21.514+08:00 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-agent-status3,ba5b112be8653986,fa92f5e335bac310] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:19:21.515422374+08:00 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:19:21.516773953+08:00 2025-10-27 11:19:21.516 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:19:21.516+08:00 2025-10-27 11:19:21.517 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:19:21.516+08:00 2025-10-27 11:19:21.519 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:19:21.519 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:19:21.518394206+08:00 2025-10-27 11:19:21.520 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:19:21.519+08:00 2025-10-27 11:19:21.523 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:19:21.523 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:19:21.514+08:00 and status AVAILABLE 2025-10-27 11:19:21.524 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:19:21.516+08:00 and status AVAILABLE 2025-10-27 11:19:21.524 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:19:21.516+08:00 and status AVAILABLE 2025-10-27 11:19:21.525 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:19:21.525 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:19:21.525 INFO 2458 --- [host-check-agent-status4,ba5b112be8653986,642bbe0635b61b92] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:19:21.527 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:19:21.519+08:00 and status AVAILABLE 2025-10-27 11:19:21.528 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:19:21.528 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:19:21.528102935+08:00 2025-10-27 11:19:21.529 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:19:21.529 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:19:21.529+08:00 2025-10-27 11:19:21.531 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:19:21.532 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:19:21.532 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:19:21.532 INFO 2458 --- [host-check-agent-status1,ba5b112be8653986,00f6b7c5ad0a3eb6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:19:21.534 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:19:21.534 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:19:21.534 INFO 2458 --- [host-check-agent-status5,ba5b112be8653986,eceaf982e1d9e5e9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:19:21.535 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:19:21.536 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:19:21.529+08:00 and status AVAILABLE 2025-10-27 11:19:21.539 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:19:21.539 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:19:21.539 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:19:21.542 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:19:21.545 INFO 2458 --- [ocp-schedules-20,fb2271cc5d8e9c56,af70f369003ca004] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:19:21.549 INFO 2458 --- [ocp-schedules-14,e21e60767a2a2c5d,848f60c5717557f6] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:19:21.550 INFO 2458 --- [ocp-schedules-20,fb2271cc5d8e9c56,af70f369003ca004] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:21.553 INFO 2458 --- [ocp-schedules-14,e21e60767a2a2c5d,848f60c5717557f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:21.553 INFO 2458 --- [ocp-schedules-14,e21e60767a2a2c5d,848f60c5717557f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:19:21.554 INFO 2458 --- [ocp-schedules-15,1083d966a67f02b9,3e456ba9d1dd6a69] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:19:21.554 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:21.557 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:21.557 INFO 2458 --- [ocp-schedules-15,1083d966a67f02b9,3e456ba9d1dd6a69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:21.558 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:21.560 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:21.560 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:21.561 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:21.562 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:21.563 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:21.563 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:21.565 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:21.566 INFO 2458 --- [ocp-schedules-23,440a674bb6539a23,b2c056555f2aa78e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:19:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:21.594 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:19:21.609 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.610 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:21.611 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:19:21.616 INFO 2458 --- [ocp-schedules-20,fb2271cc5d8e9c56,af70f369003ca004] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:19:21.622 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:21.623 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:21.624 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.624 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:21.624 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:21.625 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:21.639 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:19:21.640 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:19:21.641 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:19:21.642 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:19:21.643 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:19:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:21.684 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:19:21.684 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:19:21.684 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:19:21.685 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:19:21.686 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:19:21.696 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:19:21.716 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:21.716057+08:00, lastProcessTime=2025-10-27T11:19:11.713217+08:00 2025-10-27 11:19:21.721 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:21.721 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:21.722 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:21.723 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:21.733 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:19:21.742 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:19:21.742 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:19:21.742 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:19:21.759 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:21.760 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:19:21.761 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.762 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:19:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:21.770 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.773 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:19:21.774 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:19:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:21.779 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:19:21.783 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:19:21.784 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:19:21.796 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:19:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:21.798 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:21.809 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:21.811 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:21.812 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:21.813 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:21.826 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:19:21.827 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:19:21.830 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:19:21.831 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:21.831 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:21.835 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:19:21.855 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:21.856 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.856 INFO 2458 --- [host-check-host-status5,6b425286608b6dfe,8be72ef5b8c60c22] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:19:21.856 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:21.857 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:21.860 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.860 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:21.861 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:19:21.861 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:19:21.862 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:19:21.862 INFO 2458 --- [host-check-host-status4,6b425286608b6dfe,ab337dbbf722eacd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:19:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:21.868 INFO 2458 --- [host-check-host-status2,6b425286608b6dfe,7fd8e62eb9293a33] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:19:21.871 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:19:21.875 INFO 2458 --- [host-check-host-status3,6b425286608b6dfe,fc38ba2a0dd88b84] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:19:21.882 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:21.887 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:21.906 INFO 2458 --- [host-check-host-status1,6b425286608b6dfe,01066f39747cbefd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:19:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:21.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:19:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:21.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:21.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:21.985 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:19:21.985 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:19:21.988 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:21.989 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:19:21.992 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:21.716057+08:00 2025-10-27 11:19:22.004 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:19:22.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:22.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:22.029 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:22.030 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:22.031 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:19:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:22.052 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:19:22.052 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:19:22.100 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:22.139 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:22.139 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:22.140 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:22.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:19:22.155 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:19:22.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:22.168 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:19:22.282 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:19:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:19:22.285 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:22.286 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:19:22.318 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:19:22.354 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:22.355 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:22.355 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:19:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:19:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:19:22.393 INFO 2458 --- [ocp-schedules-15,1083d966a67f02b9,3e456ba9d1dd6a69] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:19:22.554 INFO 2458 --- [ocp-async-3,98cbe1b8ffd528ba,830f607a5c5e4f7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:22.554 WARN 2458 --- [ocp-async-4,6f1a77f71e815320,2da5b5deb85b8f31] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:23.707 INFO 2458 --- [prometheus-minute-schedule-1,e72cbf312cd397c9,721cd1842687d1e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:23.715 INFO 2458 --- [prometheus-minute-schedule-1,e72cbf312cd397c9,721cd1842687d1e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:24.240 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:24.241 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:24.506 INFO 2458 --- [ocp-schedules-20,9fa1d98e8c931133,9e35e676bc8a00a9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:24.509 INFO 2458 --- [ocp-schedules-20,9fa1d98e8c931133,9e35e676bc8a00a9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:24.554 INFO 2458 --- [ocp-async-9,abd9ae8f0a8f0ecf,1c615d1508cd9894] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:24.554 WARN 2458 --- [ocp-async-2,0acbe55126a73380,2240b29c8f059f66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:26.408 INFO 2458 --- [ocp-schedules-14,3bbcce9ccd84f8d7,9389dc751fa563df] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:26.554 INFO 2458 --- [ocp-async-7,7cfee6c3a2335e78,aabf4142974c3437] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:26.554 WARN 2458 --- [ocp-async-5,dd181ce645b50192,a58e43efff92d472] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:26.707 INFO 2458 --- [prometheus-minute-schedule-1,807d076ab0e65abe,b2b3d79a635df715] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:26.714 INFO 2458 --- [prometheus-minute-schedule-1,807d076ab0e65abe,b2b3d79a635df715] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:27.128 INFO 2458 --- [ocp-schedules-22,f6b34718e67a9a76,ad8977ee3cec615b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:19:27.136 INFO 2458 --- [ocp-schedules-22,f6b34718e67a9a76,ad8977ee3cec615b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:27.136 INFO 2458 --- [ocp-schedules-22,f6b34718e67a9a76,ad8977ee3cec615b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:27.241 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:27.242 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:27.469 INFO 2458 --- [ocp-schedules-9,ced8d6eb31d3cdaa,b1d852d4068386c3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:27.503 WARN 2458 --- [scanner-112168916-1,ced8d6eb31d3cdaa,e37f5f301270f0df] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:27.523 INFO 2458 --- [alarm-detect-18,ced8d6eb31d3cdaa,4f0c9b64284fae0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:27.528 INFO 2458 --- [alarm-detect-18,ced8d6eb31d3cdaa,4f0c9b64284fae0b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:27.538 INFO 2458 --- [alarm-detect-1,ced8d6eb31d3cdaa,6ced5e2543375b9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:27.539 INFO 2458 --- [alarm-detect-7,ced8d6eb31d3cdaa,3781dc7eedef397c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:27.543 INFO 2458 --- [alarm-detect-1,ced8d6eb31d3cdaa,6ced5e2543375b9b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:27.545 INFO 2458 --- [alarm-detect-7,ced8d6eb31d3cdaa,3781dc7eedef397c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:27.609 INFO 2458 --- [alarm-detect-10,ced8d6eb31d3cdaa,7d0693e1d89375d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:27.616 INFO 2458 --- [alarm-detect-10,ced8d6eb31d3cdaa,7d0693e1d89375d9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:27.645 INFO 2458 --- [alarm-detect-17,ced8d6eb31d3cdaa,e9106e295304c7c1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535140 ~ 1761535157 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535155, value=54.207)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535155, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535155=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:19:28.520 INFO 2458 --- [ocp-async-7,e1b2e82928696bfd,61afbc0178937dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:28.554 INFO 2458 --- [ocp-async-4,052db309e421d5c8,37772b8f02349dab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:28.554 WARN 2458 --- [ocp-async-1,255ce1e708bccfa6,86f31cad1c72d8d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:28.942 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:19:28.942 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:19:28.946 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:19:28.946 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:19:28.950 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:19:28.950 INFO 2458 --- [ocp-schedules-20,f499423fd540e974,ccd84393341d3869] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:19:29.402 INFO 2458 --- [ocp-schedules-23,f9c93a670c775647,a86496ce35f2518d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:19:29.407 INFO 2458 --- [ocp-schedules-23,f9c93a670c775647,a86496ce35f2518d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:19:29.407 INFO 2458 --- [ocp-schedules-23,f9c93a670c775647,a86496ce35f2518d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:19:29.707 INFO 2458 --- [prometheus-minute-schedule-1,3a0ac3fb767be67f,3e5912c1dfe91880] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:29.716 INFO 2458 --- [prometheus-minute-schedule-1,3a0ac3fb767be67f,3e5912c1dfe91880] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:30.242 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:30.243 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:30.554 INFO 2458 --- [ocp-async-2,c8c97e4b73d842ec,4cff26c406fca12d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:30.554 WARN 2458 --- [ocp-async-9,83b9a81cecc1c638,986a821711a2e6e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:31.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:31.715088+08:00, lastProcessTime=2025-10-27T11:19:21.716057+08:00 2025-10-27 11:19:31.925 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:31.715088+08:00 2025-10-27 11:19:32.555 WARN 2458 --- [ocp-async-7,1f77878b83704a1f,e5b174464540e43b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:32.555 INFO 2458 --- [ocp-async-6,373961b58cd3d1cf,d1e1d61bbba84150] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:32.659 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,daca7cc31cb9fa2b,e76453fb6542fbf5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=daca7cc31cb9fa2b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:19:32.659 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,daca7cc31cb9fa2b,e76453fb6542fbf5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=daca7cc31cb9fa2b, duration=0 ms] 2025-10-27 11:19:32.708 INFO 2458 --- [prometheus-minute-schedule-1,3d7d537ee5ee56a5,83c7edb28455ec94] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:32.716 INFO 2458 --- [prometheus-minute-schedule-1,3d7d537ee5ee56a5,83c7edb28455ec94] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:33.243 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:33.245 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:33.839 INFO 2458 --- [ocp-schedules-5,08e99871d9a054c1,c3d63965351958ac] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:19:33.839 INFO 2458 --- [ocp-schedules-5,08e99871d9a054c1,c3d63965351958ac] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:19:34.517 INFO 2458 --- [ocp-schedules-25,b8aa98d3b658436e,3de7c8396f607bcf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:34.520 INFO 2458 --- [ocp-schedules-25,b8aa98d3b658436e,3de7c8396f607bcf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:34.555 INFO 2458 --- [ocp-async-6,dad2e84e64e5d395,b2cf6afb1817d57c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:34.555 WARN 2458 --- [ocp-async-3,3003c96a77c65bb1,f893c53ac4d070c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:35.707 INFO 2458 --- [prometheus-minute-schedule-1,19849b24a2c8d0bb,5f79df22c2b8f5ba] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:35.716 INFO 2458 --- [prometheus-minute-schedule-1,19849b24a2c8d0bb,5f79df22c2b8f5ba] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:36.245 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:36.246 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:36.415 INFO 2458 --- [ocp-schedules-11,419948a0333fba0a,35a365193b60b426] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:36.555 INFO 2458 --- [ocp-async-6,240ac6d8cadc48c9,b44cc4f428d26491] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:36.555 WARN 2458 --- [ocp-async-9,7b49b282e0bdfbc2,cbcc8b4ba9b46cec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:37.367 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:19:37.367 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:19:37.472 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:37.473 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:19:37.485 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:37.486 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:19:37.543 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:37.544 INFO 2458 --- [ocp-schedules-19,7ea49b3fde0e0c1c,0d0ffcd33b0ccba8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:19:37.650 INFO 2458 --- [ocp-schedules-4,dbeb58d75aac5aaa,983b555ab1417853] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:37.670 WARN 2458 --- [scanner-112169170-1,dbeb58d75aac5aaa,2a4c3f3867d5df3a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:37.708 INFO 2458 --- [prometheus-cache-monitor1,fd6833bc032463e9,11256bb4d98f4d0f] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20395,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:19:37.708 INFO 2458 --- [prometheus-cache-monitor1,fd6833bc032463e9,11256bb4d98f4d0f] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:19:38.402 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:38.402 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:19:38.402 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:19:38.402 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:19:38.402 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:19:38.403 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:19:38.413 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:38.415 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:38.416 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:38.416 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:38.416 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:38.417 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:38.418 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:38.419 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:38.420 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:38.420 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:38.479 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:19:38.479 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:19:38.481 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:19:38.484 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:19:38.492 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:19:38.497 INFO 2458 --- [ocp-schedules-14,840988cf0584252b,caca5861f83eee8b] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:19:38.520 INFO 2458 --- [ocp-async-6,276696ede6101d10,03ab158db07c2065] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:38.555 INFO 2458 --- [ocp-async-2,df7feccd8aa9009d,2fefede972ae3814] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:38.555 WARN 2458 --- [ocp-async-5,cbbe0f28fca0b702,982a9399177696c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:38.683 INFO 2458 --- [ocp-schedules-15,c608ea12883ca200,fea5cebe9bdbafe0] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:19:38.689 INFO 2458 --- [ocp-schedules-15,c608ea12883ca200,fea5cebe9bdbafe0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:38.689 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:19:38.689 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:19:38.690 INFO 2458 --- [ocp-schedules-15,c608ea12883ca200,fea5cebe9bdbafe0] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:19:38.707 INFO 2458 --- [prometheus-minute-schedule-1,d49482905844bed4,6f56c395848d78fb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:38.714 INFO 2458 --- [prometheus-minute-schedule-1,d49482905844bed4,6f56c395848d78fb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:39.246 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:39.246 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:40.014 INFO 2458 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:19:40.014 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:19:40.046 INFO 2458 --- [Check obproxy1,3671cd38d2fdc274,6a96873b3fb891dc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:19:40.046 INFO 2458 --- [Check obproxy1,3671cd38d2fdc274,6a96873b3fb891dc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:19:40.050 INFO 2458 --- [Collect obproxy1,7d851d5462f99d44,7e426a5e5f296ebf] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:19:40.050 INFO 2458 --- [Collect obproxy1,7d851d5462f99d44,7e426a5e5f296ebf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:19:40.073 INFO 2458 --- [Check obproxy1,3671cd38d2fdc274,6a96873b3fb891dc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:19:40.075 INFO 2458 --- [Check obproxy1,3671cd38d2fdc274,6a96873b3fb891dc] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:19:40.075 INFO 2458 --- [Check obproxy1,3671cd38d2fdc274,6a96873b3fb891dc] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:19:40.077 INFO 2458 --- [Collect obproxy1,7d851d5462f99d44,7e426a5e5f296ebf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:19:40.080 INFO 2458 --- [Collect obproxy1,7d851d5462f99d44,7e426a5e5f296ebf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:19:40.083 INFO 2458 --- [Collect obproxy1,7d851d5462f99d44,7e426a5e5f296ebf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:19:40.555 INFO 2458 --- [ocp-async-2,11e023630d5ed625,3dd562005b26cf9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:40.555 WARN 2458 --- [ocp-async-10,1c43155d63b09578,3ad53b722ae6bdeb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:41.016 INFO 2458 --- [ocp-schedules-18,2e8fa756e8bc9b16,ce330fd6a83de0cf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:19:41.017 INFO 2458 --- [ocp-schedules-18,2e8fa756e8bc9b16,ce330fd6a83de0cf] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:19:41.707 INFO 2458 --- [prometheus-minute-schedule-1,6751953b9ed9d1a8,1693f6c71489f121] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:41.712805+08:00, lastProcessTime=2025-10-27T11:19:31.715088+08:00 2025-10-27 11:19:41.715 INFO 2458 --- [prometheus-minute-schedule-1,6751953b9ed9d1a8,1693f6c71489f121] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:41.941 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:41.712805+08:00 2025-10-27 11:19:42.247 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:42.247 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:42.555 INFO 2458 --- [ocp-async-2,e2108dc00712d997,21f78237b6891cea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:42.555 WARN 2458 --- [ocp-async-8,dcc6be877f774526,26a657eea5311667] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:44.115 INFO 2458 --- [ocp-schedules-26,ba519a9f30868359,956850d5af38d625] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:19:44.527 INFO 2458 --- [ocp-schedules-11,495e8748da996978,f7a1520ad7abd18b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:44.532 INFO 2458 --- [ocp-schedules-11,495e8748da996978,f7a1520ad7abd18b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:44.556 WARN 2458 --- [ocp-async-2,a3d0f580ed7e768f,4e47e4885303aa11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:44.556 INFO 2458 --- [ocp-async-3,4f5b8ed2d89dc870,5d6df92599060dbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:44.707 INFO 2458 --- [prometheus-minute-schedule-1,84df35317d74eccd,c749b3fb9d6114cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:44.716 INFO 2458 --- [prometheus-minute-schedule-1,84df35317d74eccd,c749b3fb9d6114cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:45.102 INFO 2458 --- [prometheus-second-schedule-1,a9e75607505321e1,c2c8ec0fa4d7ce94] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:19:45.247 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:45.249 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:46.284 INFO 2458 --- [ocp-schedules-18,53d0459bdea8b389,0c1bbb58501c75aa] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:19:46.284 INFO 2458 --- [ocp-schedules-18,53d0459bdea8b389,0c1bbb58501c75aa] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:19:46.358 INFO 2458 --- [pool-879-thread-2,53d0459bdea8b389,740b5040c403e91e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:19:46.358 INFO 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:19:46.423 INFO 2458 --- [ocp-schedules-6,f71324cdec0557c3,93cdc37118406818] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:46.424 INFO 2458 --- [pool-879-thread-2,53d0459bdea8b389,740b5040c403e91e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:46.425 INFO 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:46.427 INFO 2458 --- [pool-879-thread-2,53d0459bdea8b389,740b5040c403e91e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534886000000, 1761535186000000] 2025-10-27 11:19:46.429 INFO 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761534886000000, 1761535186000000] 2025-10-27 11:19:46.472 WARN 2458 --- [pool-879-thread-2,53d0459bdea8b389,740b5040c403e91e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:19:46.495 INFO 2458 --- [pool-879-thread-2,53d0459bdea8b389,740b5040c403e91e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:19:46.529 WARN 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:19:46.529 WARN 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:19:46.529 WARN 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:19:46.538 INFO 2458 --- [pool-879-thread-1,53d0459bdea8b389,b9d834cdd893086f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:19:46.539 INFO 2458 --- [ocp-schedules-18,53d0459bdea8b389,0c1bbb58501c75aa] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:19:46.556 INFO 2458 --- [ocp-async-10,7557b0941f4c242f,29980ef291baafcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:46.556 WARN 2458 --- [ocp-async-1,fb5ebb8e834b6c80,73d7cd1941f167b5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:47.139 INFO 2458 --- [ocp-schedules-2,47ca25888f0eadca,2e9f5ee28f96ea1d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:19:47.147 INFO 2458 --- [ocp-schedules-2,47ca25888f0eadca,2e9f5ee28f96ea1d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:47.147 INFO 2458 --- [ocp-schedules-2,47ca25888f0eadca,2e9f5ee28f96ea1d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:19:47.707 INFO 2458 --- [prometheus-minute-schedule-1,1f6d08c23464799e,2d172b8569b0deca] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:47.715 INFO 2458 --- [prometheus-minute-schedule-1,1f6d08c23464799e,2d172b8569b0deca] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:47.777 INFO 2458 --- [ocp-schedules-14,5bd885838e154417,09e56d110965cdec] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:47.830 WARN 2458 --- [scanner-112169389-1,5bd885838e154417,d4f1654be3835dca] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:47.837 INFO 2458 --- [alarm-detect-8,5bd885838e154417,3066a3504e1b365f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:47.842 INFO 2458 --- [alarm-detect-8,5bd885838e154417,3066a3504e1b365f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:19:47.855 INFO 2458 --- [alarm-detect-3,5bd885838e154417,0af91c51c8c74123] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:47.856 INFO 2458 --- [alarm-detect-1,5bd885838e154417,22c331e6ac089769] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:47.859 INFO 2458 --- [alarm-detect-3,5bd885838e154417,0af91c51c8c74123] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:19:47.860 INFO 2458 --- [alarm-detect-1,5bd885838e154417,22c331e6ac089769] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:19:47.875 INFO 2458 --- [alarm-detect-16,5bd885838e154417,1232e8bdde145e13] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535117 ~ 1761535177 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535140, value=203474.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203474.56, values={1761535140=203474.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:19:47.951 INFO 2458 --- [alarm-detect-4,5bd885838e154417,91a521b8f68b3e4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:47.956 INFO 2458 --- [alarm-detect-4,5bd885838e154417,91a521b8f68b3e4c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:19:47.964 INFO 2458 --- [alarm-detect-19,5bd885838e154417,5b908bffa6edfb22] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535117 ~ 1761535177 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535140, value=93.081)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.081, values={1761535140=93.081}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:19:47.980 INFO 2458 --- [alarm-detect-3,5bd885838e154417,d8a07a08f210eb60] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535117 ~ 1761535177 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535140, value=203475.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203475.912, values={1761535140=203475.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:19:47.988 INFO 2458 --- [alarm-detect-11,5bd885838e154417,232c3fb6c59c90c4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535155 ~ 1761535177 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535170, value=43.513)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535170, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535170=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:19:47.991 INFO 2458 --- [alarm-detect-11,5bd885838e154417,232c3fb6c59c90c4] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3, currentState=Firing 2025-10-27 11:19:47.991 INFO 2458 --- [alarm-detect-11,5bd885838e154417,232c3fb6c59c90c4] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:19:48.005 INFO 2458 --- [alarm-detect-11,5bd885838e154417,232c3fb6c59c90c4] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=353 2025-10-27 11:19:48.249 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:48.250 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:48.521 INFO 2458 --- [ocp-async-10,0bdcedbf30608f35,04c88ef46e3b5e59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:48.556 WARN 2458 --- [ocp-async-6,4560a0e43267f55f,05ac113ea8f362f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:48.556 INFO 2458 --- [ocp-async-4,918746a39ccd487d,6b78d0afcd9fb540] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:48.784 INFO 2458 --- [ocp-schedules-28,02c80c5d5586e1d2,9cb22ea7ee06e899] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:19:48.784+08:00, clusters=[1, 2] 2025-10-27 11:19:48.799 INFO 2458 --- [ocp-schedules-28,02c80c5d5586e1d2,9cb22ea7ee06e899] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:19:48.802 INFO 2458 --- [ocp-schedules-28,02c80c5d5586e1d2,9cb22ea7ee06e899] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:19:48.991 INFO 2458 --- [ocp-schedules-1,3ac4dd8741f338d4,a4409b108e35ce71] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:19:48.992 INFO 2458 --- [ocp-schedules-1,3ac4dd8741f338d4,a4409b108e35ce71] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:19:48.999 INFO 2458 --- [ocp-schedules-1,3ac4dd8741f338d4,a4409b108e35ce71] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:19:48.999 INFO 2458 --- [ocp-schedules-1,3ac4dd8741f338d4,a4409b108e35ce71] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:19:50.556 WARN 2458 --- [ocp-async-4,a56d7500f630666a,f5cd6abd95ac0ab2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:50.556 INFO 2458 --- [ocp-async-5,be25539be20dacb2,9a8bde04e2f74003] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:50.707 INFO 2458 --- [prometheus-minute-schedule-1,9d5a41a8d1f6c05c,0645b077cd7b43c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:50.715 INFO 2458 --- [prometheus-minute-schedule-1,9d5a41a8d1f6c05c,0645b077cd7b43c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:51.250 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:51.250 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:51.412 INFO 2458 --- [ocp-schedules-7,a60ef26afc3ef488,600d553b0be247fc] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:51.412 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:19:51.412 INFO 2458 --- [ocp-schedules-27,237311763459f53c,27db13020b73ee95] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:51.412 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:19:51.412 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:19:51.412 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:19:51.413 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:19:51.428 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:51.428 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:51.429 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:51.429 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:51.429 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:51.429 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:51.429 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:51.430 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:51.433 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:51.433 INFO 2458 --- [ocp-schedules-21,c69fb82115de57d0,be4de65ccb85457f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:19:51.433 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:51.434 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:51.434 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:51.434 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:51.434 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:51.435 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:51.435 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:51.435 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:51.435 INFO 2458 --- [host-check-clockdiff1,c69fb82115de57d0,8a50ac6ea26b4be0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:19:51.435 INFO 2458 --- [host-check-clockdiff2,c69fb82115de57d0,d449b11a82511f12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:19:51.437 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:51.438 INFO 2458 --- [host-check-clockdiff4,c69fb82115de57d0,1afb15277f5f7982] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:19:51.439 INFO 2458 --- [host-check-clockdiff3,c69fb82115de57d0,d8c6a029d676a09b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:19:51.440 INFO 2458 --- [host-check-clockdiff5,c69fb82115de57d0,38653a1573bad229] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:19:51.451 INFO 2458 --- [host-check-clockdiff1,c69fb82115de57d0,8a50ac6ea26b4be0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535191 0 0 , error= 2025-10-27 11:19:51.452 INFO 2458 --- [ocp-schedules-18,ba5175e97b496434,7b497c0367b2a0ea] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:19:51.471 INFO 2458 --- [host-check-clockdiff5,c69fb82115de57d0,38653a1573bad229] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535191 1 1 , error= 2025-10-27 11:19:51.482 INFO 2458 --- [host-check-clockdiff4,c69fb82115de57d0,1afb15277f5f7982] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535191 -1 -1 , error= 2025-10-27 11:19:51.484 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:51.485 INFO 2458 --- [host-check-clockdiff3,c69fb82115de57d0,d8c6a029d676a09b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535191 0 0 , error= 2025-10-27 11:19:51.488 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:51.525 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:19:51.525 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:19:51.526 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:19:51.526 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:19:51.526 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:19:51.526 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:19:51.526 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:19:51.527 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:19:51.525676277+08:00 2025-10-27 11:19:51.527 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:19:51.526229806+08:00 2025-10-27 11:19:51.527 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:19:51.527+08:00 2025-10-27 11:19:51.527 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:19:51.527+08:00 2025-10-27 11:19:51.528 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:19:51.527857694+08:00 2025-10-27 11:19:51.528 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:19:51.527467008+08:00 2025-10-27 11:19:51.528 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:19:51.528+08:00 2025-10-27 11:19:51.528 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:19:51.529 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:19:51.529 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:19:51.529 INFO 2458 --- [host-check-agent-status3,237311763459f53c,7360b8dfbd11d3e2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status4,237311763459f53c,5553428ccb912f11] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:19:51.530 INFO 2458 --- [host-check-agent-status2,237311763459f53c,22a7b17420a8abf3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:19:51.531 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:19:51.528+08:00 2025-10-27 11:19:51.531 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:19:51.532 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:19:51.532 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:19:51.532 INFO 2458 --- [host-check-agent-status1,237311763459f53c,2b683e5cd7dd6a80] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:19:51.533 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:19:51.533086005+08:00 2025-10-27 11:19:51.533 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:19:51.533+08:00 2025-10-27 11:19:51.535 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:19:51.527+08:00 and status AVAILABLE 2025-10-27 11:19:51.536 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:19:51.527+08:00 and status AVAILABLE 2025-10-27 11:19:51.537 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:19:51.528+08:00 and status AVAILABLE 2025-10-27 11:19:51.539 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:19:51.528+08:00 and status AVAILABLE 2025-10-27 11:19:51.539 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:19:51.540 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:19:51.533+08:00 and status AVAILABLE 2025-10-27 11:19:51.541 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:19:51.543 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:19:51.545 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:19:51.545 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:19:51.545 INFO 2458 --- [ocp-schedules-20,126ac703841a7df4,ae6182f4a20c335d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:19:51.549 INFO 2458 --- [ocp-schedules-12,e2b903a8043df1a1,664bb0901625461c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:19:51.550 INFO 2458 --- [ocp-schedules-20,126ac703841a7df4,ae6182f4a20c335d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:51.553 INFO 2458 --- [ocp-schedules-12,e2b903a8043df1a1,664bb0901625461c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:51.554 INFO 2458 --- [ocp-schedules-9,61091200b59709e6,6d27ef1c14d2e424] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:19:51.554 INFO 2458 --- [ocp-schedules-12,e2b903a8043df1a1,664bb0901625461c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:19:51.554 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:19:51.557 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:19:51.557 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:19:51.557 INFO 2458 --- [host-check-agent-status5,237311763459f53c,e1b2e3ea1dd107b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:19:51.561 INFO 2458 --- [ocp-schedules-9,61091200b59709e6,6d27ef1c14d2e424] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:51.564 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:19:51.564 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:19:51.564 INFO 2458 --- [ocp-schedules-25,97377162c6961f5e,2fca6b7d12e88772] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:19:51.565 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:51.567 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:19:51.567 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:19:51.567 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:51.568 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:51.568 INFO 2458 --- [ocp-schedules-30,63aa9101ea83cc16,c25e1a7d0e1263b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:19:51.569 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:19:51.570 INFO 2458 --- [ocp-schedules-26,6a72b9dd2b6ce584,9989bb0ad343e6ce] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:19:51.570 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:51.571 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:19:51.577 INFO 2458 --- [ocp-schedules-30,63aa9101ea83cc16,c25e1a7d0e1263b6] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:19:51.579 INFO 2458 --- [ocp-schedules-26,6a72b9dd2b6ce584,9989bb0ad343e6ce] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:19:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:19:51.613 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.613 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:19:51.622 INFO 2458 --- [ocp-schedules-20,126ac703841a7df4,ae6182f4a20c335d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:19:51.626 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:51.626 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.627 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:51.627 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:51.627 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:51.629 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:51.635 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:19:51.646 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:19:51.647 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:19:51.649 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:19:51.653 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:19:51.661 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:19:51.700 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:19:51.703 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.703 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:51.706 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:19:51.713 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:19:51.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:19:51.713692+08:00, lastProcessTime=2025-10-27T11:19:41.712805+08:00 2025-10-27 11:19:51.715 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:51.715 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:19:51.716 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.716 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:19:51.720 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:19:51.743 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:51.744 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:51.747 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:51.749 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:51.751 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:19:51.764 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:19:51.764 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:19:51.764 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:19:51.765 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:19:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:51.779 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:19:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.781 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:51.781 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:19:51.782 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.785 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:19:51.786 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:51.796 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:19:51.797 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:19:51.800 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:19:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:51.812 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:19:51.815 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:19:51.829 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:51.830 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:51.832 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:51.836 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:51.841 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:51.845 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:19:51.847 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:19:51.849 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:19:51.853 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:19:51.862 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:51.862 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.863 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:51.864 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:19:51.865 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:19:51.874 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:19:51.875 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:19:51.891 INFO 2458 --- [host-check-host-status4,a60ef26afc3ef488,192ff7dceb8ff16c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:19:51.892 INFO 2458 --- [host-check-host-status3,a60ef26afc3ef488,4ecd235a584ed11b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:19:51.892 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:19:51.893 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:51.895 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:19:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:19:51.897 INFO 2458 --- [host-check-host-status2,a60ef26afc3ef488,f4aeabd893b7a8dc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:19:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:51.902 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:19:51.904 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:51.914 INFO 2458 --- [host-check-host-status5,a60ef26afc3ef488,c385cf0146dcf5f8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:19:51.924 INFO 2458 --- [host-check-host-status1,a60ef26afc3ef488,cd568be80401c6d9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:19:51.964 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:51.964 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:51.978 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:19:51.989 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:19:51.713692+08:00 2025-10-27 11:19:52.007 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:52.007 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:52.019 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:19:52.020 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.020 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:19:52.022 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:19:52.043 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:19:52.045 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:52.045 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:52.067 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:19:52.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:52.082 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:19:52.091 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:19:52.091 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:19:52.118 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:52.118 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:19:52.172 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:19:52.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:52.221 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.222 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:19:52.226 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:19:52.234 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:19:52.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:19:52.247 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:19:52.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:19:52.367 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:19:52.368 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.369 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:19:52.371 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.372 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:19:52.405 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:19:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:19:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:19:52.445 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:19:52.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:19:52.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534602503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:19:52.490 INFO 2458 --- [ocp-schedules-9,61091200b59709e6,6d27ef1c14d2e424] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:19:52.556 INFO 2458 --- [ocp-async-9,a02c4b69392023be,30c5542611ca1be3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:52.556 WARN 2458 --- [ocp-async-6,8b824246e0644f0e,e46e28d6ce87038c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:19:53.225 INFO 2458 --- [host-check-clockdiff2,c69fb82115de57d0,d449b11a82511f12] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535193 -1 -1 , error= 2025-10-27 11:19:53.222 INFO 2458 --- [ocp-schedules-7,b2c71bcf7c9866a8,3bd967c5d5e5ba24] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20395,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:19:53.226 INFO 2458 --- [ocp-schedules-7,b2c71bcf7c9866a8,3bd967c5d5e5ba24] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:19:53.707 INFO 2458 --- [prometheus-minute-schedule-1,19232a4fcdd5bfdf,df723f4215d016d4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:53.715 INFO 2458 --- [prometheus-minute-schedule-1,19232a4fcdd5bfdf,df723f4215d016d4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:54.251 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:54.252 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:54.540 INFO 2458 --- [ocp-schedules-12,c340329b5f5d6ec0,09f564b44c4bd866] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:19:54.544 INFO 2458 --- [ocp-schedules-12,c340329b5f5d6ec0,09f564b44c4bd866] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:19:54.556 WARN 2458 --- [ocp-async-4,78eb8f6509b20082,94a6bfb764e0d678] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:54.556 INFO 2458 --- [ocp-async-8,f9cf538a889cfaa7,61eacd0765733f1d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:54.976 INFO 2458 --- [ocp-schedules-18,4cfd4b50bcb1adb0,5f773853cac24ec8] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:19:56.433 INFO 2458 --- [ocp-schedules-21,97b9fc0931127692,768853e1dc06314c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:19:56.556 INFO 2458 --- [ocp-async-7,37cc9a417dac65a3,eb8a99ba1c037358] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:56.557 WARN 2458 --- [ocp-async-3,9f13b5f0f275efb1,5bd95e59838ebdad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:56.707 INFO 2458 --- [prometheus-minute-schedule-1,ea45224411c2f7fb,b8dd3dc2d8cb7336] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:56.715 INFO 2458 --- [prometheus-minute-schedule-1,ea45224411c2f7fb,b8dd3dc2d8cb7336] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:19:57.252 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:19:57.253 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:19:57.998 INFO 2458 --- [ocp-schedules-16,9f1c0a93b0463689,abf38e37a4b95e98] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:19:58.006 INFO 2458 --- [ocp-schedules-16,9f1c0a93b0463689,abf38e37a4b95e98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:19:58.027 INFO 2458 --- [ocp-schedules-23,1b7d0a182577570a,0b79659193a65ae9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:19:58.073 WARN 2458 --- [scanner-112169659-1,1b7d0a182577570a,31b7a4439d924320] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:19:58.077 INFO 2458 --- [ocp-schedules-16,9f1c0a93b0463689,abf38e37a4b95e98] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:19:58.169 INFO 2458 --- [alarm-detect-3,1b7d0a182577570a,a91e31fc0c3dc48d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535170 ~ 1761535188 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535185, value=45.159)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535185, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535185=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:19:58.521 INFO 2458 --- [ocp-async-1,21a13f579660cc17,e0404b1ad7a9d656] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:19:58.557 WARN 2458 --- [ocp-async-9,a91d14b904461256,978ad80a17fd7e20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:19:58.557 INFO 2458 --- [ocp-async-6,6af739d956c97997,09a245d878234901] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:19:59.351 INFO 2458 --- [ocp-schedules-5,5c1df04bf62a3573,fb09152e77dcd821] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:19:59.707 INFO 2458 --- [prometheus-minute-schedule-1,c62c44ecc36b2985,82fcbadda5fb77d1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535200, interval=60 2025-10-27 11:19:59.716 INFO 2458 --- [prometheus-minute-schedule-1,c62c44ecc36b2985,82fcbadda5fb77d1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:00.253 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:00.254 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:00.557 WARN 2458 --- [ocp-async-2,6e8289df860ef006,19550214e020fff5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:00.557 INFO 2458 --- [ocp-async-8,edbca8fb6f6b41c1,4e40f42ba6ac80b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:00.591 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:15:01.0, nextRun=2025-10-27 11:20:00.0, currentDate=Mon Oct 27 11:20:00 CST 2025 2025-10-27 11:20:00.591 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:15:01.0, nextRunTime=2025-10-27 11:20:00.0} 2025-10-27 11:20:00.603 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:20:00.603 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:20:00.603+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:20:00.626 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:15:01.0, nextRun=2025-10-27 11:20:00.0, currentDate=Mon Oct 27 11:20:00 CST 2025 2025-10-27 11:20:00.626 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:15:01.0, nextRunTime=2025-10-27 11:20:00.0} 2025-10-27 11:20:00.717 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:20:00.724 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:20:00.731 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:20:00.732 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,ebb03d8ab0d3801d] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:20:00.732+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:20:01.266 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,148d87b28f90d3af] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9160, operation:EXECUTE, state: RUNNING 2025-10-27 11:20:01.277 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,148d87b28f90d3af] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9161, operation:EXECUTE, state: RUNNING 2025-10-27 11:20:01.299 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:20:01.300 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9160, context=Context{parallelIdx=-1, stringMap={task_instance_id=9102, task_operation=execute, latest_execution_start_time=2025-10-27T11:20:01.255+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9160}, listMap={}}, executor=10.40.65.146 2025-10-27 11:20:01.302 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:20:01.306 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9161, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9103, task_operation=execute, latest_execution_start_time=2025-10-27T11:20:01.275+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9161}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:20:01.306 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:20:01.357 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.440 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:20:01.478 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:20:01.503 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.573 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.596 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:20:01.602 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:20:01.714 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:20:01.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:01.719480+08:00, lastProcessTime=2025-10-27T11:19:51.713692+08:00 2025-10-27 11:20:01.728 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:20:01.728 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:20:01.734 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.825 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:20:01.833 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:01.835 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:20:01.861 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,487f883896139e20] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9161, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:20:02.046 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:02.099 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9103, status:SUCCESS 2025-10-27 11:20:02.112 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9103, status:SUCCESS 2025-10-27 11:20:02.130 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:20:02.152 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:20:02.153 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:20:02.161 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.183 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.198 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.224 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.311 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.359 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:01.719480+08:00 2025-10-27 11:20:02.374 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.479 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.502 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cd2e2320d2d4d01a,673c91d2a6857899] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&version=2&ObClusterId=1760510187, client=10.40.65.146, traceId=cd2e2320d2d4d01a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase,1760510187,2,{"ObRegion":"myoceanbase","ObCluster":"myoceanbase","ObRegionId":1760510187,"ObClusterId":1760510187,"Type":"PRIMARY","timestamp":1761535202501110,"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:20:02.506 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:20:02.508 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cd2e2320d2d4d01a,673c91d2a6857899] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase, obClusterId=1760510187, rootServerJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, groupName=myoceanbase, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:20:02.508 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cd2e2320d2d4d01a,673c91d2a6857899] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:20:02.508 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cd2e2320d2d4d01a,673c91d2a6857899] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase, clusterId=1760510187, affectRows=1 2025-10-27 11:20:02.508 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cd2e2320d2d4d01a,673c91d2a6857899] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.65.146, traceId=cd2e2320d2d4d01a, duration=7 ms] 2025-10-27 11:20:02.520 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:20:02.520 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:20:02.536 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,deabbcb53829896e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9160, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:20:02.557 WARN 2458 --- [ocp-async-7,3d5a8e3770bb24e2,aa8426f95d926d2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:02.557 INFO 2458 --- [ocp-async-5,15e321107a0a2d36,1be92f3e64b81fb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:02.707 INFO 2458 --- [prometheus-minute-schedule-1,f19aa97d65fe6aad,0367c5ea34a0eb09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:02.715 INFO 2458 --- [prometheus-minute-schedule-1,f19aa97d65fe6aad,0367c5ea34a0eb09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:03.136 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9102, status:SUCCESS 2025-10-27 11:20:03.148 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9102, status:SUCCESS 2025-10-27 11:20:03.254 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:03.255 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:03.446 INFO 2458 --- [ocp-schedules-4,2da8b5825461556d,e75c58348981a590] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:20:03.453 INFO 2458 --- [ocp-schedules-4,2da8b5825461556d,e75c58348981a590] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:03.492 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:03.493 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:20:03.521 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:03.521 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:20:03.535 INFO 2458 --- [ocp-schedules-4,2da8b5825461556d,e75c58348981a590] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:20:04.552 INFO 2458 --- [ocp-schedules-18,74963a607d75bd9f,692175ce6836071b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:04.556 INFO 2458 --- [ocp-schedules-18,74963a607d75bd9f,692175ce6836071b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:04.557 WARN 2458 --- [ocp-async-6,7370d217d56af7e7,2392befc71d827ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:04.557 INFO 2458 --- [ocp-async-10,ef9966f15bbac4ff,e1951dcc7d3f72d7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:04.867 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,872c1ce292137945] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:10:04.0, nextRun=2025-10-27 11:20:04.0, currentDate=Mon Oct 27 11:20:04 CST 2025 2025-10-27 11:20:04.867 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,872c1ce292137945] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-10-27 11:10:04.0, nextRunTime=2025-10-27 11:20:04.0} 2025-10-27 11:20:04.876 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,872c1ce292137945] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-10-27 11:20:04.876 INFO 2458 --- [task-coordinator1,8964d58b78e8b300,872c1ce292137945] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:20:04.876+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:20:05.311 INFO 2458 --- [subtask-coordinator3,8964d58b78e8b300,db39e93ad6eb84e8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9162, operation:EXECUTE, state: RUNNING 2025-10-27 11:20:05.342 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:20:05.343 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9162, context=Context{parallelIdx=-1, stringMap={task_instance_id=9104, task_operation=execute, latest_execution_start_time=2025-10-27T11:20:05.309+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=9162}, listMap={}}, executor=10.40.65.146 2025-10-27 11:20:05.356 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:20:05.356 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:20:05.358 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.358 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:20:05.362 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.362 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.364 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.364 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:20:05.371 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.372 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.372 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.373 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:20:05.378 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:20:05.378 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:20:05.379 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.380 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:20:05.382 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.382 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.383 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.384 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:20:05.407 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.407 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.408 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.409 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:20:05.423 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.424 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.425 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.425 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:20:05.439 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.439 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.440 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.441 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:20:05.454 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.455 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.456 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.456 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:20:05.469 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.470 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.471 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.471 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:20:05.484 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.485 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.486 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.486 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:20:05.500 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.500 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.501 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.502 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:20:05.509 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.509 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.510 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.510 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:20:05.517 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.518 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.519 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.519 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:20:05.525 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.525 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.526 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.527 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:20:05.532 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.533 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.534 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.534 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:20:05.541 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.542 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.543 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.543 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:20:05.551 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.551 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.552 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.553 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:20:05.559 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.560 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.561 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.562 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:20:05.568 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.568 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.569 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.570 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:20:05.575 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.576 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.577 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.577 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:20:05.584 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.584 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.585 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.586 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:20:05.595 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.596 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.597 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.597 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:20:05.606 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.607 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.608 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.608 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:20:05.621 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.621 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.622 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.623 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:20:05.628 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.629 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.630 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.630 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:20:05.637 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.638 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.638 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.639 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:20:05.645 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.645 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.646 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.647 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:20:05.652 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.652 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.653 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.654 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:20:05.659 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.659 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.660 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.660 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:20:05.666 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.666 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.667 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.668 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:20:05.673 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.674 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.675 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.675 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:20:05.681 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.681 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.682 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.682 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:20:05.688 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.688 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.689 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.690 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:20:05.696 INFO 2458 --- [ocp-schedules-2,543da0e13eb5f21d,bdf2332859e5f71d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:20:05.696 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.696 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.697 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.698 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:20:05.704 INFO 2458 --- [ocp-schedules-2,543da0e13eb5f21d,bdf2332859e5f71d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:05.704 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.705 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.705 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.706 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:20:05.707 INFO 2458 --- [prometheus-minute-schedule-1,25dc0f41d70626a9,49e7a535a2779d9a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:05.712 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.712 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.713 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.714 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:20:05.716 INFO 2458 --- [prometheus-minute-schedule-1,25dc0f41d70626a9,49e7a535a2779d9a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:05.723 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.723 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.724 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.727 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:20:05.733 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.733 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.734 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.735 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:20:05.743 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.743 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.744 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.745 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:20:05.751 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.751 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.752 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.753 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:20:05.759 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.761 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.761 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.762 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:20:05.767 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.768 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.769 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.769 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:20:05.775 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.775 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.776 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.777 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:20:05.782 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.783 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.784 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.784 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:20:05.790 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.790 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.791 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.792 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:20:05.798 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:20:05.798 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.800 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.800 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.801 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.801 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:20:05.802 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:20:05.808 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.809 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.809 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.810 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.810 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.810 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:20:05.815 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.816 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.817 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.817 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:20:05.822 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.823 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.824 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.824 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:20:05.830 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.830 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.831 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.832 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:20:05.836 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.837 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.837 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.838 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.838 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.839 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:20:05.842 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:20:05.844 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.845 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.845 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.846 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:20:05.851 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.851 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.852 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.852 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.854 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:20:05.857 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:20:05.859 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.860 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.861 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.861 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:20:05.863 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.863 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.865 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.866 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:05.867 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.868 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.869 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.869 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:20:05.874 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.875 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.875 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.876 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:20:05.879 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.880 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.881 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.882 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.882 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.883 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:20:05.888 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.889 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.890 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.890 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:20:05.892 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.892 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:05.894 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.895 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.896 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.897 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:20:05.902 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.903 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:05.903 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.904 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:20:05.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.908 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:20:05.922 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.923 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:05.929 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,59d45f5ae8dcc3c9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9162, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:20:05.937 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:20:05.940 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.941 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:05.949 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.950 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:05.951 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.952 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:05.953 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.954 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:05.955 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.955 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:05.956 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.957 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:05.957 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.958 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:05.959 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:20:05.961 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:20:05.972 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:05.972 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:20:06.001 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.001 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:20:06.004 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.004 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:20:06.008 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:06.009 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.009 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:20:06.019 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.019 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:20:06.019 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.019 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:20:06.022 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:20:06.022 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:20:06.036 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.037 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:20:06.050 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:20:06.070 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.070 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:20:06.073 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.073 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:20:06.096 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:20:06.101 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.102 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:20:06.124 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:20:06.125 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:20:06.125 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:20:06.125 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:20:06.128 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:20:06.129 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.130 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:20:06.132 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.133 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:20:06.134 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.135 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:20:06.182 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9104, status:SUCCESS 2025-10-27 11:20:06.183 INFO 2458 --- [task-coordinator2,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9104, status:SUCCESS 2025-10-27 11:20:06.186 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.187 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:06.187 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.188 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:06.189 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.190 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:06.190 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.191 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:06.192 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.193 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:06.193 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.194 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:06.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:06.196 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:06.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.198 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:06.199 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.199 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:20:06.200 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.200 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:20:06.201 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.202 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:20:06.202 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:20:06.205 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:20:06.232 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:17:30, start=2025-10-27T11:15:00, end=2025-10-27T11:17:30, updates=247, granule=120000ms, delay=150000ms 2025-10-27 11:20:06.246 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:17:30, target=2025-10-27T11:17:30, granule=120000ms, delay=0ms) 2025-10-27 11:20:06.255 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:06.256 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:06.287 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:06.287 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:06.288 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:20:06.312 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.312 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:20:06.312 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:20:06.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:20:06.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:20:06.313 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:20:06.316 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:20:06.316 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:20:06.316 INFO 2458 --- [ocp-schedules-2,543da0e13eb5f21d,bdf2332859e5f71d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:20:06.363 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - source=2025-10-27T11:17:30, start=2025-10-27T11:02:30, end=2025-10-27T11:12:30, updates=421, granule=600000ms, delay=900000ms 2025-10-27 11:20:06.368 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=1) - not enough cached interval (source=2025-10-27T11:17:30, target=2025-10-27T11:12:30, granule=600000ms, delay=300000ms) 2025-10-27 11:20:06.368 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=370ms, period=120000ms 2025-10-27 11:20:06.440 INFO 2458 --- [ocp-schedules-13,05ed86df953ec8f2,23e080bd6c30ce70] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:06.557 INFO 2458 --- [ocp-async-4,4e997e7950721732,27131250ca61421c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:06.557 WARN 2458 --- [ocp-async-8,f21147308101ef8c,b7a038e199c7821e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:07.149 INFO 2458 --- [ocp-schedules-8,27a74860eeadd459,88961f0c099d40dd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:20:07.157 INFO 2458 --- [ocp-schedules-8,27a74860eeadd459,88961f0c099d40dd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:07.157 INFO 2458 --- [ocp-schedules-8,27a74860eeadd459,88961f0c099d40dd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:08.175 INFO 2458 --- [ocp-schedules-25,0251fc7ea5ee6941,67346fd6a010794e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:08.218 WARN 2458 --- [scanner-112169992-1,0251fc7ea5ee6941,d7702b1d1fe0aaef] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:08.226 INFO 2458 --- [alarm-detect-10,0251fc7ea5ee6941,12a5448ce3845c10] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:08.234 INFO 2458 --- [alarm-detect-10,0251fc7ea5ee6941,12a5448ce3845c10] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:08.247 INFO 2458 --- [alarm-detect-10,0251fc7ea5ee6941,cbedc395d5cb0d3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:08.249 INFO 2458 --- [alarm-detect-9,0251fc7ea5ee6941,3ecf718f08eeb03b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:08.251 INFO 2458 --- [alarm-detect-10,0251fc7ea5ee6941,cbedc395d5cb0d3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:08.254 INFO 2458 --- [alarm-detect-9,0251fc7ea5ee6941,3ecf718f08eeb03b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:08.335 INFO 2458 --- [alarm-detect-7,0251fc7ea5ee6941,a4fe31c7de15bcec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:08.343 INFO 2458 --- [alarm-detect-7,0251fc7ea5ee6941,a4fe31c7de15bcec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:08.521 INFO 2458 --- [ocp-async-2,562d04ed24b5c386,21f166be97ce1c5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:08.557 INFO 2458 --- [ocp-async-3,e9205ecd2d6a590b,561c8efb71642845] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:08.557 WARN 2458 --- [ocp-async-1,d51ae3b3716e5104,d3ecd6712d2e49f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:08.707 INFO 2458 --- [prometheus-minute-schedule-1,a0e19a10bff21587,8d4fc5f026678f3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:08.714 INFO 2458 --- [prometheus-minute-schedule-1,a0e19a10bff21587,8d4fc5f026678f3a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:09.256 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:09.257 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:10.013 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:20:10.013 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:20:10.050 INFO 2458 --- [Check obproxy1,fefbc0631e3a08f2,6386fb04e6feabeb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:20:10.051 INFO 2458 --- [Check obproxy1,fefbc0631e3a08f2,6386fb04e6feabeb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:20:10.053 INFO 2458 --- [Collect obproxy1,5ae399bbc082ccac,be271c2ad96feb76] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:20:10.053 INFO 2458 --- [Collect obproxy1,5ae399bbc082ccac,be271c2ad96feb76] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:20:10.082 INFO 2458 --- [Collect obproxy1,5ae399bbc082ccac,be271c2ad96feb76] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:20:10.083 INFO 2458 --- [Check obproxy1,fefbc0631e3a08f2,6386fb04e6feabeb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:20:10.086 INFO 2458 --- [Collect obproxy1,5ae399bbc082ccac,be271c2ad96feb76] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:20:10.086 INFO 2458 --- [Check obproxy1,fefbc0631e3a08f2,6386fb04e6feabeb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:20:10.086 INFO 2458 --- [Check obproxy1,fefbc0631e3a08f2,6386fb04e6feabeb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:20:10.088 INFO 2458 --- [Collect obproxy1,5ae399bbc082ccac,be271c2ad96feb76] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:20:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:20:10.557 WARN 2458 --- [ocp-async-9,154af6bd6817116c,0719addb54375288] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:10.557 INFO 2458 --- [ocp-async-10,17fa1d4fa39c8c1a,733584225bccefeb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:11.104 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535140, nodeCount=47, affectRows=47 2025-10-27 11:20:11.114 INFO 2458 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535140, nodeCount=770, affectRows=770 2025-10-27 11:20:11.707 INFO 2458 --- [prometheus-minute-schedule-1,4db1cf2c255a259d,eab5cd7789496b25] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:11.715 INFO 2458 --- [prometheus-minute-schedule-1,4db1cf2c255a259d,eab5cd7789496b25] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:11.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:11.718275+08:00, lastProcessTime=2025-10-27T11:20:01.719480+08:00 2025-10-27 11:20:11.941 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:11.718275+08:00 2025-10-27 11:20:12.257 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:12.258 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:12.558 INFO 2458 --- [ocp-async-7,33de9056e2f69853,8b127eb69f8829ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:12.558 WARN 2458 --- [ocp-async-4,8963314aafcfe832,735099f20978c07f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:12.592 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=1000, affectRows=1000 2025-10-27 11:20:12.596 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=994, validCount=992 2025-10-27 11:20:12.658 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=1000, affectRows=1000 2025-10-27 11:20:12.662 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=999, validCount=995 2025-10-27 11:20:12.678 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538019) Duplicate entry '835370603008958464-1761535080' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538019) Duplicate entry '835370603008958464-1761535080' for key 'PRIMARY' 2025-10-27 11:20:12.813 ERROR 2458 --- [metric-archive-second-8,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538020) Duplicate entry '835014129044889601-1761535080' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538020) Duplicate entry '835014129044889601-1761535080' for key 'PRIMARY' 2025-10-27 11:20:12.828 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=99, affectRows=99 2025-10-27 11:20:12.828 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=97, validCount=97 2025-10-27 11:20:12.864 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=1000, affectRows=1000 2025-10-27 11:20:12.868 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=996, validCount=992 2025-10-27 11:20:14.102 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,747d30dd695fd575,39ce0240d897cc48] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=747d30dd695fd575, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:20:14.102 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,747d30dd695fd575,39ce0240d897cc48] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=747d30dd695fd575, duration=1 ms] 2025-10-27 11:20:14.558 INFO 2458 --- [ocp-async-1,02c313b1bebc1c1e,8c6b12e7a0eb58ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:14.558 WARN 2458 --- [ocp-async-5,59f70f845a3a2c68,9f6a918690f29312] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:14.564 INFO 2458 --- [ocp-schedules-6,6675265e196e9687,9070c71e6e03cddc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:14.568 INFO 2458 --- [ocp-schedules-6,6675265e196e9687,9070c71e6e03cddc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:14.707 INFO 2458 --- [prometheus-minute-schedule-1,2ce6035246e19084,a25a4227a78f0a06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:14.716 INFO 2458 --- [prometheus-minute-schedule-1,2ce6035246e19084,a25a4227a78f0a06] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:15.198 INFO 2458 --- [ocp-schedules-15,f88efa7c49faf93c,febee7f711bdc7b0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:20:15.198+08:00, clusters=[1, 2] 2025-10-27 11:20:15.199 INFO 2458 --- [ocp-schedules-15,f88efa7c49faf93c,febee7f711bdc7b0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:20:15.200 INFO 2458 --- [ocp-schedules-15,f88efa7c49faf93c,febee7f711bdc7b0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:20:15.258 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:15.259 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:16.220 INFO 2458 --- [ocp-schedules-30,1600b4b08aeb77ee,a610ae60bd3a46ed] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:20:16.220 INFO 2458 --- [ocp-schedules-30,1600b4b08aeb77ee,a610ae60bd3a46ed] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:20:16.286 INFO 2458 --- [Collect all tenants major compactions1,1600b4b08aeb77ee,abed0d6faac36c64] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:20:16.287 INFO 2458 --- [Collect all tenants major compactions2,1600b4b08aeb77ee,be0a71c548991121] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:20:16.288 INFO 2458 --- [Collect all tenants major compactions3,1600b4b08aeb77ee,fa6d3410154447ba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:20:16.288 INFO 2458 --- [Collect all tenants major compactions6,1600b4b08aeb77ee,edcba90d616bf655] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:20:16.288 INFO 2458 --- [Collect all tenants major compactions5,1600b4b08aeb77ee,3f5e6824392a42e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:20:16.288 INFO 2458 --- [Collect all tenants major compactions4,1600b4b08aeb77ee,19ef9305a956bc4e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:20:16.325 INFO 2458 --- [Collect all tenants major compactions6,1600b4b08aeb77ee,edcba90d616bf655] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.326 INFO 2458 --- [Collect all tenants major compactions6,1600b4b08aeb77ee,edcba90d616bf655] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:20:16.329 INFO 2458 --- [Collect all tenants major compactions6,1600b4b08aeb77ee,edcba90d616bf655] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:20:16.333 INFO 2458 --- [Collect all tenants major compactions5,1600b4b08aeb77ee,3f5e6824392a42e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.334 INFO 2458 --- [Collect all tenants major compactions5,1600b4b08aeb77ee,3f5e6824392a42e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:20:16.337 INFO 2458 --- [Collect all tenants major compactions5,1600b4b08aeb77ee,3f5e6824392a42e3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:20:16.348 INFO 2458 --- [Collect all tenants major compactions2,1600b4b08aeb77ee,be0a71c548991121] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.348 INFO 2458 --- [Collect all tenants major compactions3,1600b4b08aeb77ee,fa6d3410154447ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.350 INFO 2458 --- [Collect all tenants major compactions2,1600b4b08aeb77ee,be0a71c548991121] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:20:16.350 INFO 2458 --- [Collect all tenants major compactions3,1600b4b08aeb77ee,fa6d3410154447ba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:20:16.353 INFO 2458 --- [Collect all tenants major compactions3,1600b4b08aeb77ee,fa6d3410154447ba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:20:16.356 INFO 2458 --- [Collect all tenants major compactions2,1600b4b08aeb77ee,be0a71c548991121] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:20:16.359 INFO 2458 --- [Collect all tenants major compactions4,1600b4b08aeb77ee,19ef9305a956bc4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.360 INFO 2458 --- [Collect all tenants major compactions4,1600b4b08aeb77ee,19ef9305a956bc4e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:20:16.364 INFO 2458 --- [Collect all tenants major compactions1,1600b4b08aeb77ee,abed0d6faac36c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:16.364 INFO 2458 --- [Collect all tenants major compactions4,1600b4b08aeb77ee,19ef9305a956bc4e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:20:16.365 INFO 2458 --- [Collect all tenants major compactions1,1600b4b08aeb77ee,abed0d6faac36c64] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:20:16.368 INFO 2458 --- [Collect all tenants major compactions1,1600b4b08aeb77ee,abed0d6faac36c64] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:20:16.369 INFO 2458 --- [ocp-schedules-30,1600b4b08aeb77ee,a610ae60bd3a46ed] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:20:16.446 INFO 2458 --- [ocp-schedules-13,06003a17a204c6c3,27334dfab2c0231d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:16.558 WARN 2458 --- [ocp-async-9,f7cc4e947cd309bb,e68a60ce3f2807bd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:16.558 INFO 2458 --- [ocp-async-9,1977f2c9a06cb248,b731a536489e81a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:17.649 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=1000, affectRows=1000 2025-10-27 11:20:17.653 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=1000, validCount=989 2025-10-27 11:20:17.707 INFO 2458 --- [prometheus-minute-schedule-1,c9a64f9a41116d2e,cff0498636770561] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:17.714 INFO 2458 --- [prometheus-minute-schedule-1,c9a64f9a41116d2e,cff0498636770561] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:17.917 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535140, nodeCount=1000, affectRows=1000 2025-10-27 11:20:17.921 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535140, nodeCount=996, validCount=990 2025-10-27 11:20:17.933 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928} 2025-10-27 11:20:17.939 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyT 2025-10-27 11:20:17.939 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:17.947 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:17.947 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:20:17.947 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.118 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448818118000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:20:18.118Z, endTime=2025-10-27T03:18:17.928Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:20:18.187 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 260 2025-10-27 11:20:18.187 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:20:18.214 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:20:18.214 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:20:18.259 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:18.260 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:18.278 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:20:18.278 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:20:18.306 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:20:18.306 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:20:18.311 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:20:18.380 INFO 2458 --- [ocp-schedules-3,373716c8e0d87707,6b5ec854e6e3a544] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:18.402 WARN 2458 --- [scanner-112171778-1,373716c8e0d87707,d2771c76cf8ae274] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:18.415 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:20:18.118, end=2025-10-27T11:18:17.928}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:20:18.416 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:18.418 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:18.418 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:20:18.418 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.421 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:18.428 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:18.428 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:20:18.428 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.521 INFO 2458 --- [ocp-async-9,b5cbb9bf8c9727f4,0404b220eb632723] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:18.530 INFO 2458 --- [alarm-detect-20,373716c8e0d87707,456be3b3dd1349ee] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535185 ~ 1761535208 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535200, value=55.395)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535200, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535200=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:20:18.553 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448818553000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:20:18.553Z, endTime=2025-10-27T03:18:17.928Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:20:18.558 WARN 2458 --- [ocp-async-8,ac4c4360d54fc5e7,277cc32d7e200c20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:18.558 INFO 2458 --- [ocp-async-8,052196d74c2fea19,932fec2a42fbf12c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:18.570 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:20:18.570 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:20:18.595 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:20:18.595 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:20:18.601 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:20:18.601 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:20:18.628 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:20:18.629 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535097928000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:20:18.634 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:20:18.634 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:20:18.553, end=2025-10-27T11:18:17.928}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:20:18.634 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:18.636 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:18.636 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:20:18.636 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.639 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:18.640 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:18.640 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:20:18.640 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.643 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:20:18.644 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:17.258, end=2025-10-27T11:18:17.928},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:20:18.644 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:20:18.644 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:20:18.646 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:20:19.142 INFO 2458 --- [ocp-schedules-30,95544482006f8802,783118c7a6ed0ce8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:20:19.143 INFO 2458 --- [ocp-schedules-30,95544482006f8802,783118c7a6ed0ce8] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:20:19.150 INFO 2458 --- [ocp-schedules-30,95544482006f8802,783118c7a6ed0ce8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:20:19.150 INFO 2458 --- [ocp-schedules-30,95544482006f8802,783118c7a6ed0ce8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:20:19.915 INFO 2458 --- [ocp-schedules-25,0c12fd177bd28a28,52c1b033e4b28a75] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:20:20.558 INFO 2458 --- [ocp-async-6,94373f19d6349083,38d38e327498ec79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:20.558 WARN 2458 --- [ocp-async-1,04dbd11fb9f11d89,6e44585e3bdd8b6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:20.707 INFO 2458 --- [prometheus-minute-schedule-1,61b0fcec283389f0,102343fe9ba66247] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:20.714 INFO 2458 --- [prometheus-minute-schedule-1,61b0fcec283389f0,102343fe9ba66247] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:21.261 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:21.261 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:21.408 INFO 2458 --- [ocp-schedules-20,715df1374bc3ae13,d651c09cecfec397] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:21.408 INFO 2458 --- [ocp-schedules-25,7c6f797f4a0d889c,51f929b5d9176c13] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:21.408 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:20:21.409 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:20:21.409 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:20:21.409 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:20:21.409 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:20:21.410 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:20:21.410 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:20:21.410 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:20:21.410 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:20:21.410 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:20:21.423 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:21.423 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:21.426 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:21.427 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:21.427 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:21.427 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:21.428 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:21.430 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:21.430 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:21.430 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:21.431 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:21.432 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:21.432 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:21.433 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:21.433 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:21.433 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:21.433 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:21.436 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:21.438 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:21.438 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:21.439 INFO 2458 --- [ocp-schedules-2,95842b14b10d7589,b44536ccc1834095] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:21.439 INFO 2458 --- [host-check-clockdiff2,95842b14b10d7589,9fbb1b50cdbe21d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:20:21.439 INFO 2458 --- [host-check-clockdiff1,95842b14b10d7589,038e6daebe9a0060] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:20:21.440 INFO 2458 --- [host-check-clockdiff3,95842b14b10d7589,f2d63c4d171ed7bc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:20:21.440 INFO 2458 --- [host-check-clockdiff4,95842b14b10d7589,8a03dfdec12a38da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:20:21.440 INFO 2458 --- [host-check-clockdiff5,95842b14b10d7589,3068369ca5ab507b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:20:21.442 INFO 2458 --- [ocp-schedules-15,1d266976ce4bd876,6b8d2de98b8f989a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:20:21.473 INFO 2458 --- [host-check-clockdiff2,95842b14b10d7589,9fbb1b50cdbe21d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535221 -1 -1 , error= 2025-10-27 11:20:21.475 INFO 2458 --- [host-check-clockdiff1,95842b14b10d7589,038e6daebe9a0060] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535221 0 0 , error= 2025-10-27 11:20:21.483 INFO 2458 --- [host-check-clockdiff3,95842b14b10d7589,f2d63c4d171ed7bc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535221 -1 -1 , error= 2025-10-27 11:20:21.516 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:20:21.517 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:20:21.519 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:20:21.518918413+08:00 2025-10-27 11:20:21.520 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:20:21.519+08:00 2025-10-27 11:20:21.520 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:20:21.520 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:20:21.520 INFO 2458 --- [host-check-agent-status4,7c6f797f4a0d889c,8e4f8681f7b6b7d1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:20:21.522 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:20:21.523 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:20:21.524 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:20:21.523978755+08:00 2025-10-27 11:20:21.525 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:20:21.524+08:00 2025-10-27 11:20:21.526 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:20:21.526 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:20:21.527 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:20:21.527 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:20:21.527 INFO 2458 --- [host-check-agent-status3,7c6f797f4a0d889c,38812c5c23601fcd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:20:21.528 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:20:21.528158650+08:00 2025-10-27 11:20:21.528 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:20:21.527088240+08:00 2025-10-27 11:20:21.528 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:20:21.528+08:00 2025-10-27 11:20:21.528 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:20:21.528+08:00 2025-10-27 11:20:21.530 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:20:21.531 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:20:21.531 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:20:21.519+08:00 and status AVAILABLE 2025-10-27 11:20:21.532 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:20:21.531241115+08:00 2025-10-27 11:20:21.532 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:20:21.532+08:00 2025-10-27 11:20:21.532 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:20:21.532 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:20:21.532 INFO 2458 --- [host-check-agent-status5,7c6f797f4a0d889c,1d6d5f120ac85fd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:20:21.533 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:20:21.524+08:00 and status AVAILABLE 2025-10-27 11:20:21.533 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:20:21.533 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:20:21.536 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:20:21.528+08:00 and status AVAILABLE 2025-10-27 11:20:21.537 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:20:21.528+08:00 and status AVAILABLE 2025-10-27 11:20:21.537 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:20:21.537 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:20:21.537 INFO 2458 --- [host-check-agent-status2,7c6f797f4a0d889c,cbbc31fe39d7f1b2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:20:21.537 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:20:21.539 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:20:21.539 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:20:21.539 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:20:21.539 INFO 2458 --- [host-check-agent-status1,7c6f797f4a0d889c,9eac728bf8a8a123] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:20:21.540 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:20:21.532+08:00 and status AVAILABLE 2025-10-27 11:20:21.541 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:20:21.543 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:20:21.545 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:20:21.545 INFO 2458 --- [ocp-schedules-12,d34b08d543502f27,aa5692d67d21b7c0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:20:21.549 INFO 2458 --- [ocp-schedules-23,b2cc08d78f1e2c83,0ff3086018693a44] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:20:21.550 INFO 2458 --- [ocp-schedules-12,d34b08d543502f27,aa5692d67d21b7c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:21.552 INFO 2458 --- [ocp-schedules-23,b2cc08d78f1e2c83,0ff3086018693a44] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:21.553 INFO 2458 --- [ocp-schedules-23,b2cc08d78f1e2c83,0ff3086018693a44] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:20:21.554 INFO 2458 --- [ocp-schedules-21,be7a5dd7021fc3de,db905055a4e35d56] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:20:21.556 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:21.558 INFO 2458 --- [ocp-schedules-21,be7a5dd7021fc3de,db905055a4e35d56] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:21.558 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:21.559 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:21.561 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:21.562 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:21.562 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:21.565 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:21.565 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:21.567 INFO 2458 --- [ocp-schedules-29,6dc0b8ab217b880f,5d766905f4e6c257] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:20:21.568 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:21.571 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:21.587 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.588 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:21.589 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:20:21.610 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.611 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:20:21.615 INFO 2458 --- [ocp-schedules-12,d34b08d543502f27,aa5692d67d21b7c0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:20:21.618 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:21.619 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:21.624 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:21.625 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:21.632 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:21.633 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:20:21.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:21.641 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:20:21.644 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:20:21.648 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:20:21.653 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:20:21.675 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:20:21.677 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:20:21.681 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:20:21.681 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:20:21.685 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:20:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:21.699 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:21.699 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.700 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:20:21.712 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:21.712 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:21.713784+08:00, lastProcessTime=2025-10-27T11:20:11.718275+08:00 2025-10-27 11:20:21.717 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:21.722 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:21.725 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:21.728 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:20:21.733 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:20:21.734 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:20:21.746 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:20:21.758 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:20:21.759 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:21.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:21.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.764 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:20:21.765 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:20:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:20:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:21.772 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:21.773 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.774 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:20:21.787 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:20:21.790 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:21.790 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:21.794 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:21.794 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:20:21.795 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:20:21.808 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:21.809 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:20:21.819 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:21.824 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:21.824 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:21.828 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:21.858 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:20:21.858 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:20:21.858 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:20:21.858 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:20:21.863 INFO 2458 --- [host-check-host-status4,715df1374bc3ae13,f191bcbf54720ef5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:20:21.869 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:21.869 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.870 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:21.871 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:20:21.881 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:20:21.888 INFO 2458 --- [host-check-host-status2,715df1374bc3ae13,f9ed982febaf8dde] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:20:21.888 INFO 2458 --- [host-check-host-status1,715df1374bc3ae13,39edcd6ce5f75c93] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:20:21.891 INFO 2458 --- [host-check-host-status5,715df1374bc3ae13,a17891d9d5c9141d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:20:21.892 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:21.892 INFO 2458 --- [host-check-host-status3,715df1374bc3ae13,431d4948967503dd] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:20:21.895 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:21.911 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:21.912 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.913 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:21.917 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:20:21.919 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:21.962 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:20:21.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:21.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:22.004 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:20:22.005 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.005 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:20:22.006 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:21.713784+08:00 2025-10-27 11:20:22.007 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.008 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:22.008 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:20:22.008 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:22.024 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:20:22.041 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:22.041 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:22.047 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.049 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:20:22.070 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:20:22.070 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:20:22.077 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:22.077 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:22.111 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:22.111 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.201 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:20:22.215 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:20:22.224 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:22.291 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:20:22.354 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:20:22.354 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.355 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:20:22.357 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.358 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:20:22.389 INFO 2458 --- [host-check-clockdiff5,95842b14b10d7589,3068369ca5ab507b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535222 0 0 , error= 2025-10-27 11:20:22.398 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:20:22.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:22.438 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:22.439 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:20:22.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:20:22.479 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:20:22.479 INFO 2458 --- [ocp-schedules-21,be7a5dd7021fc3de,db905055a4e35d56] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:20:22.558 WARN 2458 --- [ocp-async-6,65992887555155d8,afcd61908b0a478e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:22.558 INFO 2458 --- [ocp-async-6,12b1f4e3b552bbb9,49a62e2f6aeb4998] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:23.707 INFO 2458 --- [prometheus-minute-schedule-1,e310b052ec6f3458,8357c0a1919700f6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:23.715 INFO 2458 --- [prometheus-minute-schedule-1,e310b052ec6f3458,8357c0a1919700f6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:23.970 INFO 2458 --- [host-check-clockdiff4,95842b14b10d7589,8a03dfdec12a38da] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535223 0 0 , error= 2025-10-27 11:20:24.261 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:24.262 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:24.559 WARN 2458 --- [ocp-async-7,117276e55f682811,aefd6ff024e238c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:24.559 INFO 2458 --- [ocp-async-10,decc4bed1d13ae3a,edb9d21b3d5a0da6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:24.574 INFO 2458 --- [ocp-schedules-8,553afb0caad42b4c,4750e03d1021526d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:24.578 INFO 2458 --- [ocp-schedules-8,553afb0caad42b4c,4750e03d1021526d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:26.454 INFO 2458 --- [ocp-schedules-25,e14bdacd23f32574,ddbc267a584d500f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:26.559 INFO 2458 --- [ocp-async-8,5555f2f8229a93a3,c4c3f8f7a45f8fc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:26.559 WARN 2458 --- [ocp-async-1,d99651870898079b,c80c3e48f3348275] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:26.707 INFO 2458 --- [prometheus-minute-schedule-1,b11f428e06b67aa8,55e1dc3de8f07df8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:26.714 INFO 2458 --- [prometheus-minute-schedule-1,b11f428e06b67aa8,55e1dc3de8f07df8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:27.160 INFO 2458 --- [ocp-schedules-30,5f63b7beddf6b0dd,3aab5bf74af60494] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:20:27.168 INFO 2458 --- [ocp-schedules-30,5f63b7beddf6b0dd,3aab5bf74af60494] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:27.168 INFO 2458 --- [ocp-schedules-30,5f63b7beddf6b0dd,3aab5bf74af60494] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:27.262 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:27.263 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:28.521 INFO 2458 --- [ocp-async-8,76a5244768fae88b,b4011436ed0abbca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:28.541 INFO 2458 --- [ocp-schedules-30,a6da5fc7f2a826e5,230abdbaebd96dfa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:28.554 WARN 2458 --- [scanner-112171949-1,a6da5fc7f2a826e5,1fd120d1c4e64379] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:28.559 INFO 2458 --- [ocp-async-9,feafa1b87b6f607b,8c39525c7e279702] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:28.559 WARN 2458 --- [ocp-async-3,52aae6c7074ce2bf,145264c02fe1d42a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:28.580 INFO 2458 --- [alarm-detect-11,a6da5fc7f2a826e5,8f2e457fedab82f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:28.585 INFO 2458 --- [alarm-detect-11,a6da5fc7f2a826e5,8f2e457fedab82f3] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:28.612 INFO 2458 --- [alarm-detect-17,a6da5fc7f2a826e5,61d5fad1376cc136] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:28.616 INFO 2458 --- [alarm-detect-17,a6da5fc7f2a826e5,61d5fad1376cc136] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:28.618 INFO 2458 --- [alarm-detect-5,a6da5fc7f2a826e5,f87e88b300557ddd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:28.622 INFO 2458 --- [alarm-detect-5,a6da5fc7f2a826e5,f87e88b300557ddd] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:28.687 INFO 2458 --- [alarm-detect-1,a6da5fc7f2a826e5,873e4cb465db5a6c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:28.696 INFO 2458 --- [alarm-detect-1,a6da5fc7f2a826e5,873e4cb465db5a6c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:28.727 INFO 2458 --- [alarm-detect-2,a6da5fc7f2a826e5,dcf54bea45894a73] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535200 ~ 1761535218 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535215, value=84.323)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535215, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535215=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:20:28.952 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:20:28.952 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:20:28.953 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:20:28.953 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:20:28.954 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:20:28.954 INFO 2458 --- [ocp-schedules-3,4dcf4107a11a4e8d,a69ed02e6bcebcf4] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:20:29.407 INFO 2458 --- [ocp-schedules-26,78a95eadfd64060c,1dfcc4b6b7e992b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:20:29.411 INFO 2458 --- [ocp-schedules-26,78a95eadfd64060c,1dfcc4b6b7e992b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:20:29.411 INFO 2458 --- [ocp-schedules-26,78a95eadfd64060c,1dfcc4b6b7e992b9] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:20:29.707 INFO 2458 --- [prometheus-minute-schedule-1,a7a915adb06978e1,f7b19aaf2ccda832] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:29.715 INFO 2458 --- [prometheus-minute-schedule-1,a7a915adb06978e1,f7b19aaf2ccda832] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:30.263 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:30.265 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:30.559 INFO 2458 --- [ocp-async-4,0ece09d150618136,570345e6b1da9557] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:30.559 WARN 2458 --- [ocp-async-6,55454d6e6e39840f,53f3cc086edbc5ae] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:31.019 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,8485aca2934d2959,01cc11c1101d6347] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=8485aca2934d2959, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:20:31.019 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,8485aca2934d2959,01cc11c1101d6347] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=8485aca2934d2959, duration=1 ms] 2025-10-27 11:20:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:31.714218+08:00, lastProcessTime=2025-10-27T11:20:21.713784+08:00 2025-10-27 11:20:31.917 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:31.714218+08:00 2025-10-27 11:20:32.013 INFO 2458 --- [ocp-schedules-10,a56db5d398165cef,96633358ca6e6484] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-10-27 11:20:32.013 INFO 2458 --- [ocp-schedules-10,a56db5d398165cef,96633358ca6e6484] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-10-27 11:20:32.013 INFO 2458 --- [ocp-schedules-10,a56db5d398165cef,96633358ca6e6484] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=8774 2025-10-27 11:20:32.267 INFO 2458 --- [ocp-schedules-10,a56db5d398165cef,96633358ca6e6484] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=8774 2025-10-27 11:20:32.559 WARN 2458 --- [ocp-async-5,ac8cbd5e34ad2a5c,df2919f377f01173] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:32.559 INFO 2458 --- [ocp-async-10,8f591ae0d033d204,f304213a7d936103] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:32.683 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:20:32.683 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:20:32.687 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:20:32.688 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:20:32.688 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:20:32.688 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:20:32.691 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:20:32.691 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:20:32.691 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:20:32.691 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:20:32.707 INFO 2458 --- [prometheus-minute-schedule-1,7ebf65692a25d8c2,24d877b1a4f4339d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:32.715 INFO 2458 --- [prometheus-minute-schedule-1,7ebf65692a25d8c2,24d877b1a4f4339d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:32.719 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:20:32.727 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:20:32.727 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:20:32.727 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:20:32.727 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:20:32.727 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:20:32.733 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:20:33.122 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:20:33.265 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:33.266 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:33.509 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.818087315S 2025-10-27 11:20:33.510 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:20:33.605 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:20:33.605 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:20:33.647 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:20:33.697 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:20:33.707 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:20:33.824 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] o.s.boot.SpringApplication : Started application in 1.136 seconds (JVM running for 257702.25) 2025-10-27 11:20:33.824 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.136452156S 2025-10-27 11:20:33.854 INFO 2458 --- [ocp-schedules-10,8c2e61e8c71c2d8c,b33af5ff543a9553] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:20:33.854 INFO 2458 --- [ocp-schedules-10,8c2e61e8c71c2d8c,b33af5ff543a9553] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:20:34.198 INFO 2458 --- [ocp-schedules-2,7ef28561b81079e7,1cd8184aa9a18c2a] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:20:34.559 WARN 2458 --- [ocp-async-10,30678ca8da60e874,ff0fadfe8e600f55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:34.559 INFO 2458 --- [ocp-async-2,822f8df7153f59e8,973a1cc3ea3f0520] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:34.585 INFO 2458 --- [ocp-schedules-23,68391229ffb1f34c,7de582e26145b370] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:34.589 INFO 2458 --- [ocp-schedules-23,68391229ffb1f34c,7de582e26145b370] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:35.707 INFO 2458 --- [prometheus-minute-schedule-1,66a1b87f4f97b484,25238fb1877c23df] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:35.717 INFO 2458 --- [prometheus-minute-schedule-1,66a1b87f4f97b484,25238fb1877c23df] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:36.266 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:36.267 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:36.462 INFO 2458 --- [ocp-schedules-7,6fb55996ecf132ac,7a9149fa825219db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:36.559 WARN 2458 --- [ocp-async-9,4a0b3e5a3fb426fa,5b363d62178025ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:36.559 INFO 2458 --- [ocp-async-6,663bdb9533c5cb37,a88738eaf2775a33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:37.550 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:20:37.550 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:20:37.668 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:37.669 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:20:37.681 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:37.682 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:20:37.708 INFO 2458 --- [prometheus-cache-monitor1,3425752ba0fe87b4,f6101fb268215ba7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20396,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5099,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:20:37.708 INFO 2458 --- [prometheus-cache-monitor1,3425752ba0fe87b4,f6101fb268215ba7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:20:37.744 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:37.745 INFO 2458 --- [ocp-schedules-22,40475fc849128f2c,8bb6506cc06f0be9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:20:38.509 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:38.509 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:20:38.509 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:20:38.509 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:20:38.509 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:20:38.509 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:20:38.521 INFO 2458 --- [ocp-async-6,c87e77c0aaa507c8,d7dd26607a241e20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:38.522 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:38.522 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:38.522 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:38.522 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:38.523 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:38.525 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:38.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:38.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:38.526 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:38.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:38.559 INFO 2458 --- [ocp-async-4,5d3a63a6a5920285,dd213dedfef790a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:38.559 WARN 2458 --- [ocp-async-8,7197303684b110db,fdbce159f13f1be3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:38.581 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:20:38.582 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:20:38.582 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:20:38.583 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:20:38.585 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:20:38.587 INFO 2458 --- [ocp-schedules-5,759e05de1e1b5c61,f736eec8d53dafe3] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:20:38.690 INFO 2458 --- [ocp-schedules-21,7bbd7e4fe637aba9,82499e293d4af6aa] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:20:38.697 INFO 2458 --- [ocp-schedules-21,7bbd7e4fe637aba9,82499e293d4af6aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:38.697 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:20:38.697 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:20:38.698 INFO 2458 --- [ocp-schedules-21,7bbd7e4fe637aba9,82499e293d4af6aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:20:38.707 INFO 2458 --- [prometheus-minute-schedule-1,258f380432e7fec6,c8f7515b904677f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:38.714 INFO 2458 --- [prometheus-minute-schedule-1,258f380432e7fec6,c8f7515b904677f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:38.733 INFO 2458 --- [ocp-schedules-20,c76d6a17307facf7,16072e4a76445a83] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:38.743 WARN 2458 --- [scanner-112172201-1,c76d6a17307facf7,341941b0eaee3055] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:39.267 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:39.268 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:40.013 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:20:40.013 INFO 2458 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:20:40.048 INFO 2458 --- [Check obproxy1,e153edaa71e228b6,44530c3f3233ff27] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:20:40.074 INFO 2458 --- [Check obproxy1,e153edaa71e228b6,44530c3f3233ff27] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:20:40.075 INFO 2458 --- [Collect obproxy1,887e58095ae1f722,ebe117f97f801400] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:20:40.075 INFO 2458 --- [Collect obproxy1,887e58095ae1f722,ebe117f97f801400] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:20:40.108 INFO 2458 --- [Check obproxy1,e153edaa71e228b6,44530c3f3233ff27] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:20:40.109 INFO 2458 --- [Collect obproxy1,887e58095ae1f722,ebe117f97f801400] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:20:40.111 INFO 2458 --- [Check obproxy1,e153edaa71e228b6,44530c3f3233ff27] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:20:40.112 INFO 2458 --- [Check obproxy1,e153edaa71e228b6,44530c3f3233ff27] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:20:40.113 INFO 2458 --- [Collect obproxy1,887e58095ae1f722,ebe117f97f801400] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:20:40.115 INFO 2458 --- [Collect obproxy1,887e58095ae1f722,ebe117f97f801400] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:20:40.560 INFO 2458 --- [ocp-async-3,107c8e0876b67eb9,be07834d95e05f58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:40.560 WARN 2458 --- [ocp-async-2,8b8c08aa6a334d78,2203addfb193a653] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:41.040 INFO 2458 --- [ocp-schedules-7,5dddacf43cf2668d,96b48ae35db2f663] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:20:41.042 INFO 2458 --- [ocp-schedules-7,5dddacf43cf2668d,96b48ae35db2f663] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:20:41.707 INFO 2458 --- [prometheus-minute-schedule-1,d79f4f11e26736b3,e40c4ccb5fcc7972] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:41.713516+08:00, lastProcessTime=2025-10-27T11:20:31.714218+08:00 2025-10-27 11:20:41.715 INFO 2458 --- [prometheus-minute-schedule-1,d79f4f11e26736b3,e40c4ccb5fcc7972] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:41.918 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:41.713516+08:00 2025-10-27 11:20:42.268 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:42.269 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:42.560 WARN 2458 --- [ocp-async-1,047a0b59a742db6d,c58ec3954245fd5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:42.560 INFO 2458 --- [ocp-async-7,09f4a4ed54280c48,df82ea09cf437fd1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:44.125 INFO 2458 --- [ocp-schedules-9,0f8827eadb8aca21,8040765518a54a25] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:20:44.560 INFO 2458 --- [ocp-async-6,5937207662640a06,b7c2ce907880efca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:44.560 WARN 2458 --- [ocp-async-8,91484a8acddf58e7,935ca2b5c5e3542e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:44.596 INFO 2458 --- [ocp-schedules-24,c7c54cb99d370db8,9137ce2fdd4d09db] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:44.600 INFO 2458 --- [ocp-schedules-24,c7c54cb99d370db8,9137ce2fdd4d09db] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:44.707 INFO 2458 --- [prometheus-minute-schedule-1,9fb66f5c2bed3ef2,f375fe1ae35ce0d9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:44.715 INFO 2458 --- [prometheus-minute-schedule-1,9fb66f5c2bed3ef2,f375fe1ae35ce0d9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:45.102 INFO 2458 --- [prometheus-second-schedule-1,ba79c592d69f6bf7,86b17580f8e1b036] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:20:45.269 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:45.270 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:46.473 INFO 2458 --- [ocp-schedules-17,45dab1064176bd43,8556adf750176d67] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:46.560 INFO 2458 --- [ocp-async-4,ab3607b7ad7b6a84,d077b22820fd4094] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:46.560 WARN 2458 --- [ocp-async-2,7c56b6994b90532c,fe350772c7a8c1f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:47.172 INFO 2458 --- [ocp-schedules-10,ab81dba3de6ac521,017af475b7088ea3] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:20:47.180 INFO 2458 --- [ocp-schedules-10,ab81dba3de6ac521,017af475b7088ea3] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:47.180 INFO 2458 --- [ocp-schedules-10,ab81dba3de6ac521,017af475b7088ea3] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:20:47.707 INFO 2458 --- [prometheus-minute-schedule-1,8139d3d3d9bb10c6,02ae21e417f46da4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:47.715 INFO 2458 --- [prometheus-minute-schedule-1,8139d3d3d9bb10c6,02ae21e417f46da4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:48.270 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:48.271 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:48.521 INFO 2458 --- [ocp-async-4,55b1db831d9ae926,d39a8ca6e0ae1472] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:48.560 INFO 2458 --- [ocp-async-1,8b24504da4729df2,c07a580e001f551d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:48.560 WARN 2458 --- [ocp-async-9,440a2f250deaf7dd,725f27bf06257311] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:48.840 INFO 2458 --- [ocp-schedules-17,a73cf77943941b5a,2bd678be5c62c678] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:48.860 INFO 2458 --- [ocp-schedules-14,9e1bec1bb234c674,78fae4e7eb7a05c6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:20:48.860+08:00, clusters=[1, 2] 2025-10-27 11:20:48.873 INFO 2458 --- [ocp-schedules-14,9e1bec1bb234c674,78fae4e7eb7a05c6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:20:48.876 INFO 2458 --- [ocp-schedules-14,9e1bec1bb234c674,78fae4e7eb7a05c6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:20:48.886 WARN 2458 --- [scanner-112172396-1,a73cf77943941b5a,e29a788b50087785] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:48.897 INFO 2458 --- [alarm-detect-20,a73cf77943941b5a,d7c2a9258f7d3cd9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:48.902 INFO 2458 --- [alarm-detect-20,a73cf77943941b5a,d7c2a9258f7d3cd9] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:20:48.907 INFO 2458 --- [alarm-detect-9,a73cf77943941b5a,04e2b332d82bd070] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:48.908 INFO 2458 --- [alarm-detect-5,a73cf77943941b5a,58cc72f3e8458f70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:48.912 INFO 2458 --- [alarm-detect-5,a73cf77943941b5a,58cc72f3e8458f70] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:20:48.912 INFO 2458 --- [alarm-detect-9,a73cf77943941b5a,04e2b332d82bd070] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:20:48.953 INFO 2458 --- [alarm-detect-12,a73cf77943941b5a,2d7ce154230e1086] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535178 ~ 1761535238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535200, value=203534.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203534.56, values={1761535200=203534.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:20:48.953 INFO 2458 --- [alarm-detect-12,a73cf77943941b5a,2d7ce154230e1086] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:20:48.957 INFO 2458 --- [alarm-detect-19,a73cf77943941b5a,2144d7a0353773cc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535178 ~ 1761535238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535200, value=93.091)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.091, values={1761535200=93.091}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:20:48.958 INFO 2458 --- [alarm-detect-19,a73cf77943941b5a,2144d7a0353773cc] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:20:48.985 INFO 2458 --- [alarm-detect-6,a73cf77943941b5a,7f5a8e84e00ad79b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:48.989 INFO 2458 --- [alarm-detect-6,a73cf77943941b5a,7f5a8e84e00ad79b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:20:49.025 INFO 2458 --- [alarm-detect-20,a73cf77943941b5a,47499872b489ba88] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535178 ~ 1761535238 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535200, value=203535.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203535.912, values={1761535200=203535.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:20:49.026 INFO 2458 --- [alarm-detect-19,a73cf77943941b5a,95d3253e6a80708e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535215 ~ 1761535239 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535230, value=74.205)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535230, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535230=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:20:49.026 INFO 2458 --- [alarm-detect-20,a73cf77943941b5a,47499872b489ba88] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:20:49.295 INFO 2458 --- [ocp-schedules-2,b7efcdc62b3b1303,8faff5ba6eb6efdb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:20:49.295 INFO 2458 --- [ocp-schedules-2,b7efcdc62b3b1303,8faff5ba6eb6efdb] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:20:49.302 INFO 2458 --- [ocp-schedules-2,b7efcdc62b3b1303,8faff5ba6eb6efdb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:20:49.302 INFO 2458 --- [ocp-schedules-2,b7efcdc62b3b1303,8faff5ba6eb6efdb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:20:50.560 INFO 2458 --- [ocp-async-8,1f5d397eb413bd59,de59fdb9dccb949c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:50.561 WARN 2458 --- [ocp-async-5,b41be36a5fd24e06,5a9fe6507fdd742d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:50.707 INFO 2458 --- [prometheus-minute-schedule-1,1d9d14384ce2514f,f7eb3ec397cbf7b2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:50.715 INFO 2458 --- [prometheus-minute-schedule-1,1d9d14384ce2514f,f7eb3ec397cbf7b2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:51.271 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:51.271 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:51.409 INFO 2458 --- [ocp-schedules-15,0599b4adb798a50d,91c0f699e8ea94ab] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:51.410 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:20:51.410 INFO 2458 --- [ocp-schedules-25,634f741a176c0b62,2fd3077e16739b6c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:51.410 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:20:51.410 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:20:51.410 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:20:51.411 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:20:51.424 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:51.424 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:51.425 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:51.425 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:51.425 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:51.426 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:51.426 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:51.426 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:51.429 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:51.429 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:51.430 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:51.430 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:51.431 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:51.431 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:51.431 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:51.431 INFO 2458 --- [ocp-schedules-12,81e8565fb9e04588,d7f10e3e223c4143] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:20:51.431 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:51.431 INFO 2458 --- [host-check-clockdiff2,81e8565fb9e04588,ffe83cbbb60145e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:20:51.432 INFO 2458 --- [host-check-clockdiff3,81e8565fb9e04588,42efd40bafcb7f80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:20:51.432 INFO 2458 --- [host-check-clockdiff1,81e8565fb9e04588,45a45789a87f8a26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:20:51.432 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:51.432 INFO 2458 --- [host-check-clockdiff4,81e8565fb9e04588,c3703382858d5d72] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:20:51.433 INFO 2458 --- [host-check-clockdiff5,81e8565fb9e04588,fcb786d08c6d0747] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:20:51.434 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:51.435 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:51.444 INFO 2458 --- [ocp-schedules-8,286b455c813d3638,b46d006e04b984d6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:20:51.448 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:51.457 INFO 2458 --- [host-check-clockdiff3,81e8565fb9e04588,42efd40bafcb7f80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535251 -1 -1 , error= 2025-10-27 11:20:51.457 INFO 2458 --- [host-check-clockdiff4,81e8565fb9e04588,c3703382858d5d72] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535251 -1 -1 , error= 2025-10-27 11:20:51.463 INFO 2458 --- [host-check-clockdiff5,81e8565fb9e04588,fcb786d08c6d0747] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535251 0 0 , error= 2025-10-27 11:20:51.471 INFO 2458 --- [host-check-clockdiff2,81e8565fb9e04588,ffe83cbbb60145e1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535251 -1 -1 , error= 2025-10-27 11:20:51.508 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:20:51.509 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:20:51.510 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:20:51.511 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:20:51.510401331+08:00 2025-10-27 11:20:51.512 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:20:51.512 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:20:51.513 INFO 2458 --- [host-check-agent-status2,634f741a176c0b62,abceb5a0af4190f8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:20:51.513 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:20:51.511+08:00 2025-10-27 11:20:51.513 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:20:51.513 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:20:51.513 INFO 2458 --- [host-check-agent-status1,634f741a176c0b62,19d2c9e8b5e9d8a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:20:51.514 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:20:51.517 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-agent-status3,634f741a176c0b62,2d332b1cc3e88033] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:20:51.518064740+08:00 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:20:51.518+08:00 2025-10-27 11:20:51.518 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:20:51.520 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:20:51.520698557+08:00 2025-10-27 11:20:51.521 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:20:51.520+08:00 2025-10-27 11:20:51.528 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:20:51.528 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:20:51.529 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:20:51.528503487+08:00 2025-10-27 11:20:51.529 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:20:51.529+08:00 2025-10-27 11:20:51.530 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:20:51.530169918+08:00 2025-10-27 11:20:51.531 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:20:51.531 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:20:51.531+08:00 2025-10-27 11:20:51.533 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:20:51.518+08:00 and status AVAILABLE 2025-10-27 11:20:51.533 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:20:51.533 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:20:51.534 INFO 2458 --- [host-check-agent-status5,634f741a176c0b62,59c7688490c426c3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:20:51.538 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:20:51.538 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:20:51.540 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:20:51.540 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:20:51.540 INFO 2458 --- [host-check-agent-status4,634f741a176c0b62,ccfea57421a2f659] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:20:51.541 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:20:51.511+08:00 and status AVAILABLE 2025-10-27 11:20:51.542 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:20:51.520+08:00 and status AVAILABLE 2025-10-27 11:20:51.545 INFO 2458 --- [ocp-schedules-10,03e97c0807939fc9,713b061f238f9bc6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:20:51.546 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:20:51.547 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:20:51.529+08:00 and status AVAILABLE 2025-10-27 11:20:51.548 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:20:51.548 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:20:51.531+08:00 and status AVAILABLE 2025-10-27 11:20:51.549 INFO 2458 --- [ocp-schedules-23,1e539821f9604138,fb596b320330d858] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:20:51.550 INFO 2458 --- [ocp-schedules-10,03e97c0807939fc9,713b061f238f9bc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:51.552 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:20:51.553 INFO 2458 --- [ocp-schedules-23,1e539821f9604138,fb596b320330d858] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:51.553 INFO 2458 --- [ocp-schedules-27,866a1df8c8fefb17,befe411fa7236607] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:20:51.554 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:20:51.554 INFO 2458 --- [ocp-schedules-23,1e539821f9604138,fb596b320330d858] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:20:51.558 INFO 2458 --- [ocp-schedules-27,866a1df8c8fefb17,befe411fa7236607] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:20:51.564 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:51.567 INFO 2458 --- [ocp-schedules-28,52b005a5b1ea7aa6,c80cc42c9a3055ab] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:20:51.567 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:51.569 INFO 2458 --- [ocp-schedules-17,26035ffaad655dac,5730464f20fa067c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:20:51.570 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:20:51.570 INFO 2458 --- [ocp-schedules-3,ebade63485691bf5,66f30b1b477f9f3e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:20:51.573 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:20:51.579 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:20:51.579 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:51.581 INFO 2458 --- [ocp-schedules-17,26035ffaad655dac,5730464f20fa067c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:20:51.582 INFO 2458 --- [ocp-schedules-3,ebade63485691bf5,66f30b1b477f9f3e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:20:51.583 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:20:51.583 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:51.586 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:20:51.590 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:20:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:51.602 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:20:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:51.632 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:20:51.632 INFO 2458 --- [ocp-schedules-10,03e97c0807939fc9,713b061f238f9bc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:20:51.641 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:51.657 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:51.658 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:51.664 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:51.668 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:20:51.671 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.672 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:51.676 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:20:51.679 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:20:51.680 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:20:51.681 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:20:51.688 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:20:51.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.690 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:51.711 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:20:51.713 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:20:51.717 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:20:51.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:20:51.717224+08:00, lastProcessTime=2025-10-27T11:20:41.713516+08:00 2025-10-27 11:20:51.718 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:20:51.739 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:20:51.744 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:51.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:20:51.755 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:51.757 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:51.758 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:51.759 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:51.774 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:20:51.774 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:20:51.779 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:20:51.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.780 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:20:51.781 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:20:51.786 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:20:51.791 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:20:51.813 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:20:51.819 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:20:51.820 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:20:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.830 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:20:51.831 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:20:51.831 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:51.841 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:20:51.846 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:51.850 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:51.852 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:51.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:51.856 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:51.861 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:20:51.862 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:51.870 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:20:51.878 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:20:51.883 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:20:51.887 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:20:51.890 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:51.890 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.891 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:20:51.893 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:20:51.897 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:51.901 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:51.905 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:20:51.906 INFO 2458 --- [host-check-host-status2,0599b4adb798a50d,0214fb231ceb6c79] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:20:51.906 INFO 2458 --- [host-check-host-status4,0599b4adb798a50d,c00495c3d6f6ed63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:20:51.918 INFO 2458 --- [host-check-host-status5,0599b4adb798a50d,2733d99383790374] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:20:51.926 INFO 2458 --- [host-check-host-status1,0599b4adb798a50d,622062720b126410] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:20:51.933 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:51.933 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:51.934 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:51.935 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:20:51.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:51.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:51.940 INFO 2458 --- [host-check-host-status3,0599b4adb798a50d,736c67d0660b84c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:20:51.945 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:20:51.955 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:20:51.958 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:20:51.982 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:51.983 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:52.005 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:20:51.717224+08:00 2025-10-27 11:20:52.006 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:20:52.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:52.016 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:52.052 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:20:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:52.053 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:20:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:20:52.059 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.059 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:20:52.077 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:20:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:20:52.099 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:52.104 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.104 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:20:52.126 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:20:52.127 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:20:52.127 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:20:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:20:52.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:20:52.201 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:20:52.211 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:20:52.216 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:20:52.265 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:20:52.329 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:20:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.330 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:20:52.332 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.333 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:20:52.373 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:20:52.380 INFO 2458 --- [host-check-clockdiff1,81e8565fb9e04588,45a45789a87f8a26] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535252 0 0 , error= 2025-10-27 11:20:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:20:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:20:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:20:52.457 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:20:52.457 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:20:52.457 INFO 2458 --- [ocp-schedules-27,866a1df8c8fefb17,befe411fa7236607] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:20:52.560 WARN 2458 --- [ocp-async-3,8ea3f4ff71ba4e6f,fac15b4584472e27] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:52.560 INFO 2458 --- [ocp-async-4,daee5ec3db870bdb,b7c41cda0da689fa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:20:53.707 INFO 2458 --- [prometheus-minute-schedule-1,69a4578fd2aa8f3c,72e12ebf9356c963] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:53.715 INFO 2458 --- [prometheus-minute-schedule-1,69a4578fd2aa8f3c,72e12ebf9356c963] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:54.272 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:54.272 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:54.561 INFO 2458 --- [ocp-async-9,46b11cf7b9a08811,6f41476c5e53ad01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:54.561 WARN 2458 --- [ocp-async-1,1a481f97e89cee5f,532d77c13d23ec5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:54.609 INFO 2458 --- [ocp-schedules-15,0e68839ba605840d,34109d94b0478463] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:20:54.612 INFO 2458 --- [ocp-schedules-15,0e68839ba605840d,34109d94b0478463] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:20:55.046 INFO 2458 --- [ocp-schedules-16,75afc671f8caa36f,98fdf187ffee6e60] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:20:56.482 INFO 2458 --- [ocp-schedules-25,740181c43b045325,e9664505158fdd6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:20:56.561 WARN 2458 --- [ocp-async-6,b9b00fbd725368eb,8f387f9e1288d664] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:56.561 INFO 2458 --- [ocp-async-8,ef21191447d1af1f,565e5b30cfbbe695] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:56.707 INFO 2458 --- [prometheus-minute-schedule-1,64e459c71d492138,c11a57872f68e245] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:56.715 INFO 2458 --- [prometheus-minute-schedule-1,64e459c71d492138,c11a57872f68e245] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:20:57.272 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:20:57.273 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:20:58.522 INFO 2458 --- [ocp-async-8,30f649532d65ebde,1f9669a08444c7c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:20:58.561 WARN 2458 --- [ocp-async-3,697c5e8fa1513348,d73725f385895a0a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:20:58.561 INFO 2458 --- [ocp-async-7,bdc8a465e551865a,7e5ae711b7cf1850] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:20:59.056 INFO 2458 --- [ocp-schedules-9,b4435637679a6034,fa30b6c02305957b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:20:59.091 WARN 2458 --- [scanner-112172659-1,b4435637679a6034,d4b24625c5d62353] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:20:59.187 INFO 2458 --- [alarm-detect-1,b4435637679a6034,7096932df11eaaf4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535230 ~ 1761535249 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535245, value=50.271)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535245, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535245=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:20:59.363 INFO 2458 --- [ocp-schedules-17,90c368541b505ba1,f1b4906395a1db1a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:20:59.707 INFO 2458 --- [prometheus-minute-schedule-1,b91f4ef506a2e46d,e3c36109bf9dfa4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535260, interval=60 2025-10-27 11:20:59.720 INFO 2458 --- [prometheus-minute-schedule-1,b91f4ef506a2e46d,e3c36109bf9dfa4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:00.273 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:00.274 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:00.561 WARN 2458 --- [ocp-async-1,bc15be1b634a9658,89ac27bfd05ea2a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:00.561 INFO 2458 --- [ocp-async-5,832a2c7008c456cc,ef2bb617461d1e46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:01.714189+08:00, lastProcessTime=2025-10-27T11:20:51.717224+08:00 2025-10-27 11:21:01.938 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:01.714189+08:00 2025-10-27 11:21:02.561 WARN 2458 --- [ocp-async-5,1415923a09781886,1c20ad1a95dd2cb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:02.561 INFO 2458 --- [ocp-async-6,1ef3e244b6f0481c,1cb2a8f57f393eb6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:02.707 INFO 2458 --- [prometheus-minute-schedule-1,479846d44d0cbcd8,b865726b0e12a5a6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:02.717 INFO 2458 --- [prometheus-minute-schedule-1,479846d44d0cbcd8,b865726b0e12a5a6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:03.274 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:03.275 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:03.536 INFO 2458 --- [ocp-schedules-23,b058442764e60250,1e15ce894eacf32b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:21:03.543 INFO 2458 --- [ocp-schedules-23,b058442764e60250,1e15ce894eacf32b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:03.582 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:03.583 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:21:03.595 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:03.595 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:21:03.615 INFO 2458 --- [ocp-schedules-23,b058442764e60250,1e15ce894eacf32b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:21:04.561 INFO 2458 --- [ocp-async-10,2314b0fbd49a40dc,daa64798ca0fea33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:04.561 WARN 2458 --- [ocp-async-3,fbec3e3ab2a247bb,1058887706118c22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:04.620 INFO 2458 --- [ocp-schedules-17,0d853ba6179673fa,4e4e86ae1a4833fa] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:04.623 INFO 2458 --- [ocp-schedules-17,0d853ba6179673fa,4e4e86ae1a4833fa] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:05.707 INFO 2458 --- [prometheus-minute-schedule-1,eba4c55ab25b1c39,3dd5e48be2f1f232] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:05.715 INFO 2458 --- [prometheus-minute-schedule-1,eba4c55ab25b1c39,3dd5e48be2f1f232] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:06.275 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:06.276 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:06.317 INFO 2458 --- [ocp-schedules-15,dde752e9e3b4df35,0610b9b992065748] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:21:06.321 INFO 2458 --- [ocp-schedules-15,dde752e9e3b4df35,0610b9b992065748] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:06.407 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:21:06.408 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.411 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:21:06.422 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.423 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.448 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.448 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.459 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:21:06.459 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:21:06.477 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.477 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.478 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.478 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.479 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.480 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.499 INFO 2458 --- [ocp-schedules-11,e5206d8dac002263,69e11a04a5fbe444] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:06.506 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.506 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.509 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.540 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.541 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.541 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.541 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:21:06.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.561 WARN 2458 --- [ocp-async-10,f530a438780417d0,c232e8b338d20617] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:06.561 INFO 2458 --- [ocp-async-1,c4020d3344e1b9b2,b3d120d7e8a22feb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:06.571 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.575 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.576 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.577 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.578 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.578 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.579 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.580 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.581 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.581 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.582 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.583 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.584 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.584 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:21:06.590 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:21:06.599 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.600 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:21:06.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:21:06.632 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.632 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:21:06.641 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:06.641 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.642 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:21:06.653 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.653 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:21:06.653 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.653 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:21:06.656 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:21:06.656 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:21:06.662 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.662 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:21:06.665 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.665 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:21:06.694 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.695 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:21:06.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.721 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:21:06.767 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.767 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.768 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.769 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.769 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.770 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.771 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.772 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.772 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.773 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.773 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.774 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.775 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.775 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.776 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.777 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.778 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.778 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.779 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.780 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:21:06.780 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.781 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:21:06.782 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.782 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:21:06.783 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:21:06.785 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:21:06.863 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:06.863 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:06.864 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:21:06.888 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:21:06.892 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:21:06.892 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:21:06.893 INFO 2458 --- [ocp-schedules-15,dde752e9e3b4df35,0610b9b992065748] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:21:07.182 INFO 2458 --- [ocp-schedules-19,f777a9731b2e5788,b2f93e133b87e88d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:21:07.190 INFO 2458 --- [ocp-schedules-19,f777a9731b2e5788,b2f93e133b87e88d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:07.190 INFO 2458 --- [ocp-schedules-19,f777a9731b2e5788,b2f93e133b87e88d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:07.984 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:21:07.984+08:00, tenantSize=6 2025-10-27 11:21:08.024 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761535267984000 2025-10-27 11:21:08.024 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761534967924000 2025-10-27 11:21:08.065 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:21:08.076 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761535268070000 2025-10-27 11:21:08.076 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761534967857000 2025-10-27 11:21:08.098 INFO 2458 --- [ocp-schedules-11,cbb6602f3276e854,794beceeb4980061] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:21:08.522 INFO 2458 --- [ocp-async-1,2885377a6712729d,c15e1eadc7434f7a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:08.562 WARN 2458 --- [ocp-async-8,823a1fede9e63047,f2bc00ca052490c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:08.562 INFO 2458 --- [ocp-async-6,3bbb09f449b7d0b1,055639fccd2864f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:08.707 INFO 2458 --- [prometheus-minute-schedule-1,a65a38a624ad16e7,a8301194b2d545bf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:08.716 INFO 2458 --- [prometheus-minute-schedule-1,a65a38a624ad16e7,a8301194b2d545bf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:09.192 INFO 2458 --- [ocp-schedules-2,65170149b5d7a096,7dca9c4e7d1d7589] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:21:09.200 WARN 2458 --- [scanner-112172826-1,65170149b5d7a096,168dba88a3cd7681] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:21:09.244 INFO 2458 --- [alarm-detect-4,65170149b5d7a096,13c579dbaeafe9a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:09.251 INFO 2458 --- [alarm-detect-9,65170149b5d7a096,3666bfe54433ab1f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:09.251 INFO 2458 --- [alarm-detect-4,65170149b5d7a096,13c579dbaeafe9a6] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:09.252 INFO 2458 --- [alarm-detect-5,65170149b5d7a096,d3bc925f7104fa99] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:09.255 INFO 2458 --- [alarm-detect-9,65170149b5d7a096,3666bfe54433ab1f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:09.257 INFO 2458 --- [alarm-detect-5,65170149b5d7a096,d3bc925f7104fa99] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:09.276 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:09.277 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:09.318 INFO 2458 --- [alarm-detect-20,65170149b5d7a096,deff7c2aaf05ac4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:09.322 INFO 2458 --- [alarm-detect-20,65170149b5d7a096,deff7c2aaf05ac4d] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:10.012 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:21:10.012 INFO 2458 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:21:10.050 INFO 2458 --- [Check obproxy1,9dc909aac914d3e7,421bf2c9abe33d59] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:21:10.050 INFO 2458 --- [Check obproxy1,9dc909aac914d3e7,421bf2c9abe33d59] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:21:10.050 INFO 2458 --- [Collect obproxy1,69721f303badf90f,99b2ce63aa92e1c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:21:10.050 INFO 2458 --- [Collect obproxy1,69721f303badf90f,99b2ce63aa92e1c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:21:10.080 INFO 2458 --- [Collect obproxy1,69721f303badf90f,99b2ce63aa92e1c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:21:10.082 INFO 2458 --- [Check obproxy1,9dc909aac914d3e7,421bf2c9abe33d59] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:21:10.083 INFO 2458 --- [Collect obproxy1,69721f303badf90f,99b2ce63aa92e1c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:21:10.085 INFO 2458 --- [Check obproxy1,9dc909aac914d3e7,421bf2c9abe33d59] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:21:10.085 INFO 2458 --- [Check obproxy1,9dc909aac914d3e7,421bf2c9abe33d59] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:21:10.086 INFO 2458 --- [Collect obproxy1,69721f303badf90f,99b2ce63aa92e1c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:21:10.186 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:21:10.562 WARN 2458 --- [ocp-async-4,8644e2327302fd93,fa53a5fb615b153b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:10.562 INFO 2458 --- [ocp-async-6,aaca1e33464cde23,a35344ff5dbf82ed] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:11.145 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535200, nodeCount=47, affectRows=47 2025-10-27 11:21:11.169 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535200, nodeCount=47, affectRows=47 2025-10-27 11:21:11.707 INFO 2458 --- [prometheus-minute-schedule-1,fca3bf86f76419c9,718f32a412508dcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:11.713447+08:00, lastProcessTime=2025-10-27T11:21:01.714189+08:00 2025-10-27 11:21:11.715 INFO 2458 --- [prometheus-minute-schedule-1,fca3bf86f76419c9,718f32a412508dcb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:11.730 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=1000, affectRows=1000 2025-10-27 11:21:11.734 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=994, validCount=992 2025-10-27 11:21:11.789 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=1000, affectRows=1000 2025-10-27 11:21:11.793 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=999, validCount=995 2025-10-27 11:21:11.817 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134601996) Duplicate entry '835014129065861120-1761535140' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134601996) Duplicate entry '835014129065861120-1761535140' for key 'PRIMARY' 2025-10-27 11:21:11.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:11.713447+08:00 2025-10-27 11:21:11.936 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134538020) Duplicate entry '835370602870546432-1761535140' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134538020) Duplicate entry '835370602870546432-1761535140' for key 'PRIMARY' 2025-10-27 11:21:12.277 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:12.278 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:12.475 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,beea447edfe0a2e6,2a379816dde9c4b0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=beea447edfe0a2e6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:21:12.476 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,beea447edfe0a2e6,2a379816dde9c4b0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=beea447edfe0a2e6, duration=2 ms] 2025-10-27 11:21:12.562 INFO 2458 --- [ocp-async-6,fc13b886d2823234,8746279674ba0889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:12.562 WARN 2458 --- [ocp-async-5,b281e7ee76fb0f25,2dd40f9e5b1b0d15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:12.882 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=99, affectRows=99 2025-10-27 11:21:12.883 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=98, validCount=98 2025-10-27 11:21:12.922 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=1000, affectRows=1000 2025-10-27 11:21:12.926 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=1000, validCount=994 2025-10-27 11:21:13.859 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535200, nodeCount=727, affectRows=727 2025-10-27 11:21:14.562 WARN 2458 --- [ocp-async-6,d571eaee5c33f71f,092372c43fb5d5a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:14.562 INFO 2458 --- [ocp-async-8,84b77fbf6b8d934d,bbefa17d82210b53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:14.630 INFO 2458 --- [ocp-schedules-6,1e0443df7b6551d8,ac1360b09d82b275] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:14.634 INFO 2458 --- [ocp-schedules-6,1e0443df7b6551d8,ac1360b09d82b275] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:14.707 INFO 2458 --- [prometheus-minute-schedule-1,1ed8dbe8665cad01,1f1913ae12e57f4d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:14.715 INFO 2458 --- [prometheus-minute-schedule-1,1ed8dbe8665cad01,1f1913ae12e57f4d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:15.266 INFO 2458 --- [ocp-schedules-24,df54a110b2b2bb02,41eee739daef34a7] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:21:15.266+08:00, clusters=[1, 2] 2025-10-27 11:21:15.270 INFO 2458 --- [ocp-schedules-24,df54a110b2b2bb02,41eee739daef34a7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:21:15.270 INFO 2458 --- [ocp-schedules-24,df54a110b2b2bb02,41eee739daef34a7] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:21:15.278 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:15.279 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:16.370 INFO 2458 --- [ocp-schedules-9,17ee93fae9d1779b,0f4a6806787f53e4] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:21:16.370 INFO 2458 --- [ocp-schedules-9,17ee93fae9d1779b,0f4a6806787f53e4] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:21:16.436 INFO 2458 --- [Collect all tenants major compactions1,17ee93fae9d1779b,d647fdd23a9d5dc8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:21:16.436 INFO 2458 --- [Collect all tenants major compactions2,17ee93fae9d1779b,4b67b941cb1b2f82] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:21:16.437 INFO 2458 --- [Collect all tenants major compactions3,17ee93fae9d1779b,cf8bb4ecea609e31] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:21:16.437 INFO 2458 --- [Collect all tenants major compactions5,17ee93fae9d1779b,163d50f0dce985f2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:21:16.437 INFO 2458 --- [Collect all tenants major compactions4,17ee93fae9d1779b,5053189c2742a59f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:21:16.437 INFO 2458 --- [Collect all tenants major compactions6,17ee93fae9d1779b,0cda0dd750fc2768] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:21:16.475 INFO 2458 --- [Collect all tenants major compactions5,17ee93fae9d1779b,163d50f0dce985f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.475 INFO 2458 --- [Collect all tenants major compactions6,17ee93fae9d1779b,0cda0dd750fc2768] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.477 INFO 2458 --- [Collect all tenants major compactions5,17ee93fae9d1779b,163d50f0dce985f2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:21:16.477 INFO 2458 --- [Collect all tenants major compactions6,17ee93fae9d1779b,0cda0dd750fc2768] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:21:16.484 INFO 2458 --- [Collect all tenants major compactions5,17ee93fae9d1779b,163d50f0dce985f2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:21:16.487 INFO 2458 --- [Collect all tenants major compactions6,17ee93fae9d1779b,0cda0dd750fc2768] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:21:16.495 INFO 2458 --- [Collect all tenants major compactions4,17ee93fae9d1779b,5053189c2742a59f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.495 INFO 2458 --- [Collect all tenants major compactions2,17ee93fae9d1779b,4b67b941cb1b2f82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.495 INFO 2458 --- [Collect all tenants major compactions3,17ee93fae9d1779b,cf8bb4ecea609e31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.497 INFO 2458 --- [Collect all tenants major compactions4,17ee93fae9d1779b,5053189c2742a59f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:21:16.498 INFO 2458 --- [Collect all tenants major compactions2,17ee93fae9d1779b,4b67b941cb1b2f82] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:21:16.501 INFO 2458 --- [Collect all tenants major compactions2,17ee93fae9d1779b,4b67b941cb1b2f82] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:21:16.501 INFO 2458 --- [Collect all tenants major compactions4,17ee93fae9d1779b,5053189c2742a59f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:21:16.502 INFO 2458 --- [Collect all tenants major compactions3,17ee93fae9d1779b,cf8bb4ecea609e31] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:21:16.503 INFO 2458 --- [Collect all tenants major compactions1,17ee93fae9d1779b,d647fdd23a9d5dc8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:16.505 INFO 2458 --- [ocp-schedules-19,c55af35f444ca6b7,9e3af6c575c2537e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:16.507 INFO 2458 --- [Collect all tenants major compactions3,17ee93fae9d1779b,cf8bb4ecea609e31] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:21:16.507 INFO 2458 --- [Collect all tenants major compactions1,17ee93fae9d1779b,d647fdd23a9d5dc8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:21:16.510 INFO 2458 --- [Collect all tenants major compactions1,17ee93fae9d1779b,d647fdd23a9d5dc8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:21:16.511 INFO 2458 --- [ocp-schedules-9,17ee93fae9d1779b,0f4a6806787f53e4] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:21:16.562 INFO 2458 --- [ocp-async-8,f0dea54b44e59a93,1856d19e6c09d08e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:16.562 WARN 2458 --- [ocp-async-7,7871ed81188dcd95,9f1df7a9dc716e85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:16.853 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=1000, affectRows=1000 2025-10-27 11:21:16.857 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=999, validCount=997 2025-10-27 11:21:16.957 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535200, nodeCount=1000, affectRows=1000 2025-10-27 11:21:16.962 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535200, nodeCount=992, validCount=979 2025-10-27 11:21:17.707 INFO 2458 --- [prometheus-minute-schedule-1,1a02e682ab9a53a3,62a55bbb60ebcfbf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:17.715 INFO 2458 --- [prometheus-minute-schedule-1,1a02e682ab9a53a3,62a55bbb60ebcfbf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:18.248 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.256 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpCluste 2025-10-27 11:21:18.280 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:18.281 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:18.293 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.304 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.310 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.522 INFO 2458 --- [ocp-async-10,ea1246e2590abe07,38f0aaefbf24f3da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:18.562 INFO 2458 --- [ocp-async-5,ec70f7281b23e2bd,5b744f93101762f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:18.562 WARN 2458 --- [ocp-async-1,40915cdb35a5763b,881f4734cad73e8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:18.607 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.609 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.771 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.773 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.777 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.778 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.780 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.781 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.782 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.783 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.784 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.785 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.786 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.788 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.789 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.790 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.791 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.795 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.798 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.800 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.803 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:15:18.018, end=2025-10-27T11:20:18.243} 2025-10-27 11:21:18.803 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:21:19.344 INFO 2458 --- [ocp-schedules-29,df4a45f04944e99b,30214faac94fb7bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:21:19.390 WARN 2458 --- [scanner-112174644-1,df4a45f04944e99b,a2d68adf629f4ab5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:21:19.479 INFO 2458 --- [ocp-schedules-30,6f6230533bf789dc,fb94c71a745cb118] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:21:19.479 INFO 2458 --- [ocp-schedules-30,6f6230533bf789dc,fb94c71a745cb118] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:21:19.489 INFO 2458 --- [ocp-schedules-30,6f6230533bf789dc,fb94c71a745cb118] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:21:19.489 INFO 2458 --- [ocp-schedules-30,6f6230533bf789dc,fb94c71a745cb118] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:21:19.511 INFO 2458 --- [alarm-detect-2,df4a45f04944e99b,993f2f6926e4397e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535245 ~ 1761535269 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535260, value=49.485)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535260, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535260=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:21:19.967 INFO 2458 --- [ocp-schedules-9,d0384bde8d89223d,4f8845c5c0b189f6] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:21:20.563 INFO 2458 --- [ocp-async-2,1f154c5194464aea,dc2b83b8443ef139] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:20.563 WARN 2458 --- [ocp-async-3,06350e4bfdd0b3fc,b066f3a58d12389a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:20.707 INFO 2458 --- [prometheus-minute-schedule-1,07e628f1e4740a3d,0792502883b5c764] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:20.721 INFO 2458 --- [prometheus-minute-schedule-1,07e628f1e4740a3d,0792502883b5c764] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:21.281 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:21.282 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:21.409 INFO 2458 --- [ocp-schedules-29,662187f31e231cca,ab801da3400d6101] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:21.409 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:21:21.409 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:21:21.409 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:21:21.409 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:21:21.409 INFO 2458 --- [ocp-schedules-9,4e3efc75a6669fad,882bdf6f3d5da024] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:21.409 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:21:21.410 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:21:21.410 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:21:21.410 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:21:21.410 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:21:21.410 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:21:21.423 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:21.424 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:21.424 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:21.424 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:21.424 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:21.424 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:21.425 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:21.426 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:21.426 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:21.429 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:21.429 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:21.429 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:21.429 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:21.429 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:21.430 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:21.430 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:21.430 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:21.435 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:21.435 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:21.436 INFO 2458 --- [ocp-schedules-27,e8b974a7efd37f65,294041d9fe7e3f41] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:21.437 INFO 2458 --- [host-check-clockdiff2,e8b974a7efd37f65,d1332e40c04d5b56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:21:21.437 INFO 2458 --- [host-check-clockdiff1,e8b974a7efd37f65,e3a6a3ab126c0e9a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:21:21.437 INFO 2458 --- [host-check-clockdiff3,e8b974a7efd37f65,7c838c09f3d75388] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:21:21.438 INFO 2458 --- [host-check-clockdiff4,e8b974a7efd37f65,77f672490c0c0310] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:21:21.438 INFO 2458 --- [host-check-clockdiff5,e8b974a7efd37f65,e3df634dac15b1d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:21:21.438 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:21.442 INFO 2458 --- [ocp-schedules-11,49e15bf300db8607,81ffc132d710248d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:21:21.450 INFO 2458 --- [host-check-clockdiff2,e8b974a7efd37f65,d1332e40c04d5b56] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535281 -1 -1 , error= 2025-10-27 11:21:21.456 INFO 2458 --- [host-check-clockdiff3,e8b974a7efd37f65,7c838c09f3d75388] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535281 0 0 , error= 2025-10-27 11:21:21.462 INFO 2458 --- [host-check-clockdiff5,e8b974a7efd37f65,e3df634dac15b1d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535281 1 1 , error= 2025-10-27 11:21:21.464 INFO 2458 --- [host-check-clockdiff1,e8b974a7efd37f65,e3a6a3ab126c0e9a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535281 0 0 , error= 2025-10-27 11:21:21.528 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:21:21.530 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:21:21.529086651+08:00 2025-10-27 11:21:21.535 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:21:21.530+08:00 2025-10-27 11:21:21.544 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:21:21.530+08:00 and status AVAILABLE 2025-10-27 11:21:21.545 INFO 2458 --- [ocp-schedules-15,9bc0410a5ce6a4e3,c08fcfcba47d9117] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:21:21.548 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:21:21.549 INFO 2458 --- [ocp-schedules-25,c4515ab52e01595d,2f9585b75346bbfa] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:21:21.550 INFO 2458 --- [ocp-schedules-15,9bc0410a5ce6a4e3,c08fcfcba47d9117] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:21.553 INFO 2458 --- [ocp-schedules-25,c4515ab52e01595d,2f9585b75346bbfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:21.554 INFO 2458 --- [ocp-schedules-5,f9317611fe1a6fd2,c24a882495b4f3a8] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:21:21.554 INFO 2458 --- [ocp-schedules-25,c4515ab52e01595d,2f9585b75346bbfa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:21:21.558 INFO 2458 --- [ocp-schedules-5,f9317611fe1a6fd2,c24a882495b4f3a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:21.579 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:21:21.581 INFO 2458 --- [ocp-schedules-1,7e77ce4fa7e22858,994f2f1dde393973] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:21:21.582 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:21:21.582 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:21:21.582 INFO 2458 --- [host-check-agent-status3,4e3efc75a6669fad,9a1b572434ea63b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:21:21.594 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:21.600 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:21.607 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:21:21.607 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:21:21.607 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:21:21.610 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:21:21.611 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:21:21.611 INFO 2458 --- [host-check-agent-status4,4e3efc75a6669fad,0d45336f3f2a7a91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:21:21.611 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:21:21.611 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:21:21.611 INFO 2458 --- [host-check-agent-status5,4e3efc75a6669fad,4c7a3c8c2727a015] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:21:21.612 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:21:21.612 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:21:21.612 INFO 2458 --- [host-check-agent-status1,4e3efc75a6669fad,f6cc614b49f2923e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:21:21.613 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:21:21.613 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:21:21.613151959+08:00 2025-10-27 11:21:21.613 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:21:21.614 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:21:21.613+08:00 2025-10-27 11:21:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.615 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:21:21.614395974+08:00 2025-10-27 11:21:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:21.616 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:21:21.615+08:00 2025-10-27 11:21:21.616 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:21:21.623 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:21:21.613+08:00 and status AVAILABLE 2025-10-27 11:21:21.624 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:21:21.615+08:00 and status AVAILABLE 2025-10-27 11:21:21.630 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:21:21.630 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:21:21.630906415+08:00 2025-10-27 11:21:21.631 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:21:21.630+08:00 2025-10-27 11:21:21.631 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:21:21.633 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:21:21.633 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:21:21.636 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:21:21.636 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:21:21.636 INFO 2458 --- [host-check-agent-status2,4e3efc75a6669fad,181094ff545900fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:21:21.638 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:21.639 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:21:21.630+08:00 and status AVAILABLE 2025-10-27 11:21:21.644 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:21:21.646 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:21:21.647 INFO 2458 --- [ocp-schedules-15,9bc0410a5ce6a4e3,c08fcfcba47d9117] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:21:21.653 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:21:21.654 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:21:21.654321383+08:00 2025-10-27 11:21:21.655 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:21:21.654+08:00 2025-10-27 11:21:21.658 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:21.660 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:21.661 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:21.666 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:21.666 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.667 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:21.667 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:21:21.654+08:00 and status AVAILABLE 2025-10-27 11:21:21.669 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:21.669 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:21.671 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:21.672 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:21:21.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:21.692 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:21:21.701 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:21.704 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:21.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:21.716807+08:00, lastProcessTime=2025-10-27T11:21:11.713447+08:00 2025-10-27 11:21:21.724 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:21.729 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:21:21.729 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:21.734 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:21.744 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:21:21.746 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:21:21.755 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:21:21.758 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:21.759 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.760 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:21:21.767 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:21.776 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:21.777 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:21:21.779 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:21.779 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:21:21.780 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.780 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:21:21.783 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:21:21.800 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:21:21.803 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:21:21.828 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:21.832 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:21:21.835 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:21.838 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:21.838 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:21.841 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.843 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:21:21.844 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:21.846 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:21:21.852 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:21:21.853 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:21:21.854 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:21:21.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:21.869 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:21.877 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:21.880 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:21:21.881 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:21:21.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:21.885 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:21:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:21.888 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:21.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.892 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:21:21.893 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:21.905 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:21:21.908 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:21:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:21.923 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:21.924 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:21.935 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:21:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:21.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:21.937 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:21.938 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:21:21.940 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:21.940 INFO 2458 --- [host-check-host-status3,662187f31e231cca,996f1805ad040523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:21:21.941 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:21.941 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:21.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:21:21.946 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:21:21.954 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:21:21.956 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:21:21.968 INFO 2458 --- [host-check-host-status4,662187f31e231cca,c9938cad928f235b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:21:21.970 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:21.972 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:21.973 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:21.974 INFO 2458 --- [host-check-host-status2,662187f31e231cca,fa984715f7f110c6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:21:21.983 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:21.983 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:21.991 INFO 2458 --- [host-check-host-status5,662187f31e231cca,0fc93c7bb55d1164] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:21:22.003 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:21:22.020 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:22.020 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:22.032 INFO 2458 --- [host-check-host-status1,662187f31e231cca,92c5013b64a76902] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:21:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:21:22.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:22.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:22.075 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:21:22.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:21:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.079 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:21:22.101 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:21:22.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:22.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:22.123 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:22.124 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.124 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:21:22.147 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:22.151 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:21:22.151 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761534684669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:21:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.191 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:22.196 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:21:22.208 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:21:22.222 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:22.226 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:21:22.348 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:21:22.349 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.350 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:21:22.352 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.353 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:21:22.369 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:21.716807+08:00 2025-10-27 11:21:22.393 INFO 2458 --- [host-check-clockdiff4,e8b974a7efd37f65,77f672490c0c0310] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535282 0 0 , error= 2025-10-27 11:21:22.402 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:21:22.444 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:22.445 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:22.445 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:21:22.486 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:21:22.486 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:21:22.486 INFO 2458 --- [ocp-schedules-5,f9317611fe1a6fd2,c24a882495b4f3a8] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:21:22.563 WARN 2458 --- [ocp-async-4,8f8d269db6bbd311,3eb3acb40f44a2e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:22.563 INFO 2458 --- [ocp-async-8,1dab64ef3a37c32c,61de7ea4235058ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:23.707 INFO 2458 --- [prometheus-minute-schedule-1,d8203c31b817562b,0dc1ac43ac39c318] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:23.716 INFO 2458 --- [prometheus-minute-schedule-1,d8203c31b817562b,0dc1ac43ac39c318] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:24.282 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:24.283 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:24.563 WARN 2458 --- [ocp-async-10,f3328fc80f452b5a,cb1ab2054b7237e1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:24.563 INFO 2458 --- [ocp-async-9,944a9d4dd7ec5b6a,9e16c2b8eab9d98a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:24.641 INFO 2458 --- [ocp-schedules-2,5577d107f4c3bc8a,8179cc703b91d1c4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:24.645 INFO 2458 --- [ocp-schedules-2,5577d107f4c3bc8a,8179cc703b91d1c4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:24.667 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,50341c6275b580fc,3a0a64c19b5265d6] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&version=2&ObClusterId=1760510188, client=10.40.66.107, traceId=50341c6275b580fc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase2,1760510188,2,{"ObRegion":"myoceanbase2","ObCluster":"myoceanbase2","ObRegionId":1760510188,"ObClusterId":1760510188,"Type":"PRIMARY","timestamp":1761535284666582,"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:21:24.673 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,50341c6275b580fc,3a0a64c19b5265d6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase2, obClusterId=1760510188, rootServerJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, groupName=myoceanbase2, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:21:24.673 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,50341c6275b580fc,3a0a64c19b5265d6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:21:24.673 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,50341c6275b580fc,3a0a64c19b5265d6] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase2, clusterId=1760510188, affectRows=1 2025-10-27 11:21:24.673 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,50341c6275b580fc,3a0a64c19b5265d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.66.107, traceId=50341c6275b580fc, duration=6 ms] 2025-10-27 11:21:26.511 INFO 2458 --- [ocp-schedules-7,5cff1adf881790a1,6d161e93f1cedbe4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:26.563 INFO 2458 --- [ocp-async-5,f84ed018b49adf4b,29fe538eab5642b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:26.563 WARN 2458 --- [ocp-async-6,4c4dee91a990748a,f59e96db8e2d8f36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:26.707 INFO 2458 --- [prometheus-minute-schedule-1,ac7f76c711d772df,6220e4169d50879b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:26.714 INFO 2458 --- [prometheus-minute-schedule-1,ac7f76c711d772df,6220e4169d50879b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:27.194 INFO 2458 --- [ocp-schedules-25,9af242bcefe6d07f,8ccf0ab0414739cf] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:21:27.203 INFO 2458 --- [ocp-schedules-25,9af242bcefe6d07f,8ccf0ab0414739cf] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:27.203 INFO 2458 --- [ocp-schedules-25,9af242bcefe6d07f,8ccf0ab0414739cf] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:27.283 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:27.284 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:28.522 INFO 2458 --- [ocp-async-7,f00badbddcfa3515,d5d98f1af7a92df7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:28.563 WARN 2458 --- [ocp-async-2,2a0d6d7eb4f4ae0c,8e62ce1dfe2df026] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:28.563 INFO 2458 --- [ocp-async-4,5ccb44175ba269a5,0fc55a648c3d164f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:28.955 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:21:28.955 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:21:28.960 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:21:28.960 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:21:28.963 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:21:28.963 INFO 2458 --- [ocp-schedules-17,8412bd135a18037f,6ec563dabc7631ae] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:21:29.377 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5844da2fd4d5cdd8,e01bd617610c7215] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=5844da2fd4d5cdd8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:21:29.377 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,5844da2fd4d5cdd8,e01bd617610c7215] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=5844da2fd4d5cdd8, duration=1 ms] 2025-10-27 11:21:29.379 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,d36ce2d6ff8aae29,97e54998fd248919] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest, client=10.40.67.22, traceId=d36ce2d6ff8aae29, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,false,] 2025-10-27 11:21:29.379 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,d36ce2d6ff8aae29,97e54998fd248919] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=d36ce2d6ff8aae29, duration=1 ms] 2025-10-27 11:21:29.411 INFO 2458 --- [ocp-schedules-26,d211d39f50dafc02,a3d856c190734de6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:21:29.415 INFO 2458 --- [ocp-schedules-26,d211d39f50dafc02,a3d856c190734de6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:21:29.415 INFO 2458 --- [ocp-schedules-26,d211d39f50dafc02,a3d856c190734de6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:21:29.518 INFO 2458 --- [ocp-schedules-13,b2dc946d68a6ed8c,3d5d5823eaf09412] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:21:29.562 WARN 2458 --- [scanner-112174809-1,b2dc946d68a6ed8c,8227fe1fe8adb062] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:21:29.571 INFO 2458 --- [alarm-detect-7,b2dc946d68a6ed8c,2acaeaf18617a597] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:29.576 INFO 2458 --- [alarm-detect-7,b2dc946d68a6ed8c,2acaeaf18617a597] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:29.586 INFO 2458 --- [alarm-detect-19,b2dc946d68a6ed8c,bb39ee22ee0adff7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:29.588 INFO 2458 --- [alarm-detect-7,b2dc946d68a6ed8c,54e6f34fdada9314] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:29.591 INFO 2458 --- [alarm-detect-19,b2dc946d68a6ed8c,bb39ee22ee0adff7] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:29.593 INFO 2458 --- [alarm-detect-7,b2dc946d68a6ed8c,54e6f34fdada9314] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:29.655 INFO 2458 --- [alarm-detect-18,b2dc946d68a6ed8c,20fa1178afdd4e0a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:29.660 INFO 2458 --- [alarm-detect-18,b2dc946d68a6ed8c,20fa1178afdd4e0a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:29.703 INFO 2458 --- [alarm-detect-5,b2dc946d68a6ed8c,bf5c2aaf226f692e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535260 ~ 1761535279 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535275, value=52.165)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535275, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535275=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:21:29.707 INFO 2458 --- [prometheus-minute-schedule-1,0f0907aec54c3eb5,53262cdf1056dc0f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:29.714 INFO 2458 --- [prometheus-minute-schedule-1,0f0907aec54c3eb5,53262cdf1056dc0f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:30.284 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:30.285 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:30.563 WARN 2458 --- [ocp-async-10,5dc7ac47b282ec07,ff6be29790d3f750] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:30.563 INFO 2458 --- [ocp-async-10,cb6a333eeacd1275,5105e5046b4a2d54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:31.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:31.713235+08:00, lastProcessTime=2025-10-27T11:21:21.716807+08:00 2025-10-27 11:21:31.931 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:31.713235+08:00 2025-10-27 11:21:32.563 WARN 2458 --- [ocp-async-6,e93e95b47fb2605e,875e3e95cb0671d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:32.563 INFO 2458 --- [ocp-async-3,eab28d9a614b6064,70fd9c7d5c429e84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:32.707 INFO 2458 --- [prometheus-minute-schedule-1,72f479bc00d82bd7,548f5def465ac031] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:32.715 INFO 2458 --- [prometheus-minute-schedule-1,72f479bc00d82bd7,548f5def465ac031] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:33.285 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:33.287 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:33.862 INFO 2458 --- [ocp-schedules-2,3b542e5763cc3225,06a44c8f3c08df84] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:21:33.862 INFO 2458 --- [ocp-schedules-2,3b542e5763cc3225,06a44c8f3c08df84] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:21:34.235 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:21:34.235 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:21:34.253 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:21:34.253 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:21:34.253 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:21:34.265 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:21:34.265 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:21:34.265 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:21:34.280 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:21:34.280 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:21:34.280 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:21:34.293 INFO 2458 --- [ocp-schedules-10,2b2ab17bec3f63eb,96dda3258858d517] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:21:34.563 WARN 2458 --- [ocp-async-7,57f665a27aad6fa8,c32cf7c579bf9f46] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:34.563 INFO 2458 --- [ocp-async-7,63e81ed6dd27f1e6,63babd28814c10a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:34.652 INFO 2458 --- [ocp-schedules-17,378682ff387afd2a,5f56fa696d9a9e87] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:34.656 INFO 2458 --- [ocp-schedules-17,378682ff387afd2a,5f56fa696d9a9e87] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:35.707 INFO 2458 --- [prometheus-minute-schedule-1,ce69865548b59467,0574d2ac4a77b8ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:35.717 INFO 2458 --- [prometheus-minute-schedule-1,ce69865548b59467,0574d2ac4a77b8ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:36.221 INFO 2458 --- [ocp-schedules-8,667021ae2b42f02e,00a5bc6d9083a392] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-10-27 11:21:36.232 INFO 2458 --- [ocp-schedules-8,667021ae2b42f02e,00a5bc6d9083a392] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:36.287 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:36.288 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:36.324 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:36.328 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 02:04:47.657729, 2025-10-27 11:21:36.278, 1000] 2025-10-27 11:21:36.328 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2025-10-27 02:04:47.657729, endTime 2025-10-27 11:21:36.278 2025-10-27 11:21:36.340 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:36.344 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091391, 2025-10-27 11:21:36.278, 1000] 2025-10-27 11:21:36.345 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2025-10-27 08:00:44.091391, endTime 2025-10-27 11:21:36.278 2025-10-27 11:21:36.416 INFO 2458 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:36.421 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:36.439 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:36.439 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:36.443 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-17 15:45:19.719943, 2025-10-27 11:21:36.376, 1000] 2025-10-27 11:21:36.444 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2025-10-17 15:45:19.719943, endTime 2025-10-27 11:21:36.376 2025-10-27 11:21:36.461 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091611, 2025-10-27 11:21:36.38, 1000] 2025-10-27 11:21:36.462 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2025-10-27 08:00:44.091611, endTime 2025-10-27 11:21:36.38 2025-10-27 11:21:36.462 INFO 2458 --- [ocp-schedules-8,667021ae2b42f02e,00a5bc6d9083a392] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=2, successCount=2 2025-10-27 11:21:36.522 INFO 2458 --- [ocp-schedules-30,396beb0b07a53a2d,9e7a40249da949dc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:36.564 WARN 2458 --- [ocp-async-4,3c6d7c8a2d8a55a3,b10f67929e1fece6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:36.564 INFO 2458 --- [ocp-async-4,0cb675b0f1a66982,92fdf33c5941f9e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:37.708 INFO 2458 --- [prometheus-cache-monitor1,d9e11824018e790c,6a536f4cfc3a8e61] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20396,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5096,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:21:37.708 INFO 2458 --- [prometheus-cache-monitor1,d9e11824018e790c,6a536f4cfc3a8e61] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:21:37.750 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:21:37.750 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:21:37.857 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:37.858 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:21:37.869 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:37.870 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:21:37.929 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:37.930 INFO 2458 --- [ocp-schedules-5,b81e56b18bc8bf3c,8219043e726ec095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:21:38.522 INFO 2458 --- [ocp-async-4,8337de80c8751368,dc006b618819578a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:38.564 WARN 2458 --- [ocp-async-5,8854a3aa6ba416a2,41921d8a2b9bd7cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:38.564 INFO 2458 --- [ocp-async-6,a7c480a971a3afc8,9e968f79a1dd9193] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:38.600 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:38.600 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:21:38.600 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:21:38.600 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:21:38.600 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:21:38.600 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:21:38.611 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:38.611 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:38.612 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:38.612 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:38.613 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:38.615 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:38.616 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:38.616 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:38.616 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:38.617 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:38.677 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:21:38.678 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:21:38.678 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:21:38.679 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:21:38.680 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:21:38.682 INFO 2458 --- [ocp-schedules-3,ae8e7af6f4bdbc89,3c194d1193dc1974] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:21:38.698 INFO 2458 --- [ocp-schedules-25,74994ab7f6f7f996,eb7afdada1b9cc1b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:21:38.702 INFO 2458 --- [ocp-schedules-25,74994ab7f6f7f996,eb7afdada1b9cc1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:38.702 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:21:38.702 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:21:38.703 INFO 2458 --- [ocp-schedules-25,74994ab7f6f7f996,eb7afdada1b9cc1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:21:38.707 INFO 2458 --- [prometheus-minute-schedule-1,32473a7738b0f9c5,12805017b08d4070] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:38.718 INFO 2458 --- [prometheus-minute-schedule-1,32473a7738b0f9c5,12805017b08d4070] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:39.288 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:39.289 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:39.708 INFO 2458 --- [ocp-schedules-8,75674e28c801d791,c237eb1f57688a04] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:21:39.740 WARN 2458 --- [scanner-112175072-1,75674e28c801d791,a52cbea994ce58fe] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:21:40.013 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:21:40.013 INFO 2458 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:21:40.048 INFO 2458 --- [Collect obproxy1,f3e202587ff50433,742ba91717d21767] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:21:40.048 INFO 2458 --- [Collect obproxy1,f3e202587ff50433,742ba91717d21767] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:21:40.048 INFO 2458 --- [Check obproxy1,720e6df06a600e77,47ecc4f22d1423be] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:21:40.048 INFO 2458 --- [Check obproxy1,720e6df06a600e77,47ecc4f22d1423be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:21:40.078 INFO 2458 --- [Collect obproxy1,f3e202587ff50433,742ba91717d21767] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:21:40.078 INFO 2458 --- [Check obproxy1,720e6df06a600e77,47ecc4f22d1423be] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:21:40.081 INFO 2458 --- [Collect obproxy1,f3e202587ff50433,742ba91717d21767] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:21:40.081 INFO 2458 --- [Check obproxy1,720e6df06a600e77,47ecc4f22d1423be] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:21:40.081 INFO 2458 --- [Check obproxy1,720e6df06a600e77,47ecc4f22d1423be] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:21:40.083 INFO 2458 --- [Collect obproxy1,f3e202587ff50433,742ba91717d21767] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:21:40.564 WARN 2458 --- [ocp-async-8,31a1d91e92ad000b,e07d899fcf581e42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:40.564 INFO 2458 --- [ocp-async-7,e8d51d2a357c24a4,0314702a234b2456] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:41.063 INFO 2458 --- [ocp-schedules-7,7046a04d60a4890a,fdb651f7ba3a5e46] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:21:41.064 INFO 2458 --- [ocp-schedules-7,7046a04d60a4890a,fdb651f7ba3a5e46] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:21:41.707 INFO 2458 --- [prometheus-minute-schedule-1,b815946d5798593a,5d61120839a2704b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:41.713 INFO 2458 --- [prometheus-minute-schedule-1,b815946d5798593a,5d61120839a2704b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:41.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:41.717027+08:00, lastProcessTime=2025-10-27T11:21:31.713235+08:00 2025-10-27 11:21:41.940 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:41.717027+08:00 2025-10-27 11:21:42.289 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:42.291 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:42.564 WARN 2458 --- [ocp-async-1,832d54202168ba6c,2fd85465d3e2d52c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:42.564 INFO 2458 --- [ocp-async-9,87f1b827fd5acd44,957958f331f61f08] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:44.135 INFO 2458 --- [ocp-schedules-22,373b00d1f7b3ecc7,ceb67680654461f0] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:21:44.564 WARN 2458 --- [ocp-async-5,7d9b73b926a0397b,e36efbf183c9d417] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:44.564 INFO 2458 --- [ocp-async-4,5e4d406c86798556,e6e6f03489203182] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:44.666 INFO 2458 --- [ocp-schedules-10,2444c37f6c811ed4,c6a902fb73495160] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:44.669 INFO 2458 --- [ocp-schedules-10,2444c37f6c811ed4,c6a902fb73495160] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:44.707 INFO 2458 --- [prometheus-minute-schedule-1,6b6b2b31c84507c9,b656878e20f54965] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:44.719 INFO 2458 --- [prometheus-minute-schedule-1,6b6b2b31c84507c9,b656878e20f54965] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:45.102 INFO 2458 --- [prometheus-second-schedule-1,8bd12f623a7b286d,f2d19894909895a7] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:21:45.291 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:45.292 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:46.530 INFO 2458 --- [ocp-schedules-9,a6c77f56b6d06d72,b441824737c08090] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:46.564 WARN 2458 --- [ocp-async-6,8bf1ed389aff62df,c0f96dbe15881ce0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:46.564 INFO 2458 --- [ocp-async-6,9093a83b1a6aaad5,d9bae491ea5e7728] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:47.206 INFO 2458 --- [ocp-schedules-6,2a49673b759917b1,223468ae1f60af27] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:21:47.213 INFO 2458 --- [ocp-schedules-6,2a49673b759917b1,223468ae1f60af27] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:47.213 INFO 2458 --- [ocp-schedules-6,2a49673b759917b1,223468ae1f60af27] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:21:47.707 INFO 2458 --- [prometheus-minute-schedule-1,7146a82f5db09481,a567b732ac25701f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:47.715 INFO 2458 --- [prometheus-minute-schedule-1,7146a82f5db09481,a567b732ac25701f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:48.292 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:48.294 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:48.522 INFO 2458 --- [ocp-async-6,ba96078235cff428,05d7d51ae7b521d6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:48.565 WARN 2458 --- [ocp-async-1,c3cbf17ff608ec3b,7326d5046e3add13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:48.565 INFO 2458 --- [ocp-async-10,11f9bd724be49e26,eb9540cc823302e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:48.932 INFO 2458 --- [ocp-schedules-1,d10594321a60af69,4f0fe16bd5ac38db] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:21:48.932+08:00, clusters=[1, 2] 2025-10-27 11:21:48.948 INFO 2458 --- [ocp-schedules-1,d10594321a60af69,4f0fe16bd5ac38db] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:21:48.950 INFO 2458 --- [ocp-schedules-1,d10594321a60af69,4f0fe16bd5ac38db] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:21:49.637 INFO 2458 --- [ocp-schedules-13,dd6c0f4d9d7ae38c,3e6afaa28ad36eda] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:21:49.637 INFO 2458 --- [ocp-schedules-13,dd6c0f4d9d7ae38c,3e6afaa28ad36eda] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:21:49.644 INFO 2458 --- [ocp-schedules-13,dd6c0f4d9d7ae38c,3e6afaa28ad36eda] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:21:49.645 INFO 2458 --- [ocp-schedules-13,dd6c0f4d9d7ae38c,3e6afaa28ad36eda] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:21:49.852 INFO 2458 --- [ocp-schedules-25,576b830aa030b687,5e09a7e36b88fe1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:21:49.913 WARN 2458 --- [scanner-112175271-1,576b830aa030b687,d59c64564657de81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:21:49.918 INFO 2458 --- [alarm-detect-11,576b830aa030b687,830a0f807e31e386] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:49.922 INFO 2458 --- [alarm-detect-11,576b830aa030b687,830a0f807e31e386] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:21:49.934 INFO 2458 --- [alarm-detect-17,576b830aa030b687,e50f2e35a1a9f389] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535239 ~ 1761535299 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535260, value=203594.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203594.56, values={1761535260=203594.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:21:49.939 INFO 2458 --- [alarm-detect-10,576b830aa030b687,a00f91266c4cc88f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:49.940 INFO 2458 --- [alarm-detect-4,576b830aa030b687,466523c6f2eed00e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:49.944 INFO 2458 --- [alarm-detect-4,576b830aa030b687,466523c6f2eed00e] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:21:49.945 INFO 2458 --- [alarm-detect-10,576b830aa030b687,a00f91266c4cc88f] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:21:50.012 INFO 2458 --- [alarm-detect-16,576b830aa030b687,432e3cc37ed5b663] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535239 ~ 1761535299 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535260, value=93.1)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.1, values={1761535260=93.1}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:21:50.015 INFO 2458 --- [alarm-detect-4,576b830aa030b687,52900cc8b784e436] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535239 ~ 1761535299 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535260, value=203595.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203595.912, values={1761535260=203595.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:21:50.016 INFO 2458 --- [alarm-detect-19,576b830aa030b687,9bc55a4e60e5915a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:50.020 INFO 2458 --- [alarm-detect-19,576b830aa030b687,9bc55a4e60e5915a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:21:50.064 INFO 2458 --- [alarm-detect-4,576b830aa030b687,68959d3763a2842e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535275 ~ 1761535300 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535290, value=45.784)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535290, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535290=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:21:50.565 WARN 2458 --- [ocp-async-5,bbee9b73e4256b00,b8d15b64a219892f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:50.565 INFO 2458 --- [ocp-async-3,7ffc71a2cdda32b6,d280bc1eabd27850] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:50.707 INFO 2458 --- [prometheus-minute-schedule-1,e86bbdabdb742cb9,7c662e5b2163fd16] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:50.722 INFO 2458 --- [prometheus-minute-schedule-1,e86bbdabdb742cb9,7c662e5b2163fd16] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:51.294 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:51.295 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:51.410 INFO 2458 --- [ocp-schedules-28,8a7d0020c497a07c,907539f1d46ac234] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:51.410 INFO 2458 --- [ocp-schedules-23,be5da83e5f87f0ae,fff4f812aaf6946e] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:51.410 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:21:51.410 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:21:51.410 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:21:51.410 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:21:51.411 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:21:51.426 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:51.426 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:51.426 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:51.426 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:51.427 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:51.427 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:51.427 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:51.427 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:51.431 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:51.431 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:51.431 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:51.431 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:51.432 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:51.433 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:51.433 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:51.433 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:51.434 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:51.434 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:51.435 INFO 2458 --- [ocp-schedules-17,e9feeeb2f9d5b6ba,546f6e157fcf743f] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:21:51.436 INFO 2458 --- [host-check-clockdiff1,e9feeeb2f9d5b6ba,72097a3d2bc6d848] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:21:51.437 INFO 2458 --- [host-check-clockdiff2,e9feeeb2f9d5b6ba,b3e415f7d37aab3a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:21:51.437 INFO 2458 --- [host-check-clockdiff3,e9feeeb2f9d5b6ba,c902c53365f6c023] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:21:51.438 INFO 2458 --- [host-check-clockdiff4,e9feeeb2f9d5b6ba,dd6e669587226813] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:21:51.438 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:51.438 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:51.438 INFO 2458 --- [host-check-clockdiff5,e9feeeb2f9d5b6ba,881eddc959fdfa9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:21:51.448 INFO 2458 --- [ocp-schedules-29,dfb4454c7a39cd9d,38cebac2ee6b2728] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:21:51.462 INFO 2458 --- [host-check-clockdiff2,e9feeeb2f9d5b6ba,b3e415f7d37aab3a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535311 -1 -1 , error= 2025-10-27 11:21:51.463 INFO 2458 --- [host-check-clockdiff4,e9feeeb2f9d5b6ba,dd6e669587226813] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535311 0 0 , error= 2025-10-27 11:21:51.465 INFO 2458 --- [host-check-clockdiff5,e9feeeb2f9d5b6ba,881eddc959fdfa9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535311 0 0 , error= 2025-10-27 11:21:51.474 INFO 2458 --- [host-check-clockdiff1,e9feeeb2f9d5b6ba,72097a3d2bc6d848] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535311 0 0 , error= 2025-10-27 11:21:51.500 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:21:51.502 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:21:51.501181375+08:00 2025-10-27 11:21:51.502 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:21:51.502+08:00 2025-10-27 11:21:51.504 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:21:51.504 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:21:51.506 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-agent-status5,be5da83e5f87f0ae,384d26e4ecc05010] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:21:51.508 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:21:51.509 INFO 2458 --- [host-check-agent-status1,be5da83e5f87f0ae,479edb9a5ac58d48] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:21:51.509 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:21:51.509068352+08:00 2025-10-27 11:21:51.510 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:21:51.509+08:00 2025-10-27 11:21:51.510 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:21:51.502+08:00 and status AVAILABLE 2025-10-27 11:21:51.510 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:21:51.510 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:21:51.510 INFO 2458 --- [host-check-agent-status2,be5da83e5f87f0ae,2f43eee02c75becf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:21:51.511 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:21:51.512 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:21:51.512 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:21:51.513 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:21:51.512849431+08:00 2025-10-27 11:21:51.514 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:21:51.513+08:00 2025-10-27 11:21:51.514 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:21:51.514 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:21:51.514 INFO 2458 --- [host-check-agent-status4,be5da83e5f87f0ae,b6a7e93bc991fb95] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:21:51.515 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:21:51.515 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:21:51.515 INFO 2458 --- [host-check-agent-status3,be5da83e5f87f0ae,e21786550a805a6d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:21:51.516 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:21:51.516 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:21:51.517 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:21:51.515349237+08:00 2025-10-27 11:21:51.517 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:21:51.517 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:21:51.517+08:00 2025-10-27 11:21:51.518 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:21:51.518203553+08:00 2025-10-27 11:21:51.518 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:21:51.509+08:00 and status AVAILABLE 2025-10-27 11:21:51.518 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:21:51.518+08:00 2025-10-27 11:21:51.521 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:21:51.513+08:00 and status AVAILABLE 2025-10-27 11:21:51.523 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:21:51.524 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:21:51.517+08:00 and status AVAILABLE 2025-10-27 11:21:51.525 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:21:51.526 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:21:51.518+08:00 and status AVAILABLE 2025-10-27 11:21:51.529 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:21:51.531 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:21:51.536 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:21:51.539 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:21:51.543 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:51.544 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:21:51.545 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:51.545 INFO 2458 --- [ocp-schedules-18,e3960e8b163cc513,262af04dc901aaea] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:21:51.546 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:21:51.548 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:21:51.549 INFO 2458 --- [ocp-schedules-8,b9a952c56b37d07b,e872ba0ef9c73d46] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:21:51.550 INFO 2458 --- [ocp-schedules-18,e3960e8b163cc513,262af04dc901aaea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:51.550 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:21:51.551 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:51.552 INFO 2458 --- [ocp-schedules-8,b9a952c56b37d07b,e872ba0ef9c73d46] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:51.553 INFO 2458 --- [ocp-schedules-8,b9a952c56b37d07b,e872ba0ef9c73d46] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:21:51.553 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:51.554 INFO 2458 --- [ocp-schedules-1,1e4f3e04206f948e,a6f4c453622c07b9] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:21:51.559 INFO 2458 --- [ocp-schedules-1,1e4f3e04206f948e,a6f4c453622c07b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:51.567 INFO 2458 --- [ocp-schedules-14,4ea3ed11acc67ea8,b6d31910c463fab1] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:21:51.568 INFO 2458 --- [ocp-schedules-19,7ef8afd258d34098,d09b2d08c8383452] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:21:51.569 INFO 2458 --- [ocp-schedules-26,c462f1d365a1a701,e46948f8da781f89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:21:51.578 INFO 2458 --- [ocp-schedules-26,c462f1d365a1a701,e46948f8da781f89] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:21:51.582 INFO 2458 --- [ocp-schedules-19,7ef8afd258d34098,d09b2d08c8383452] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:21:51.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.598 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:51.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:21:51.609 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:51.613 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:51.614 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:21:51.614 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:51.614 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.615 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:51.617 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:21:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:21:51.619 INFO 2458 --- [ocp-schedules-18,e3960e8b163cc513,262af04dc901aaea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:21:51.629 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:21:51.633 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:21:51.635 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:21:51.640 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:21:51.641 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.641 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:51.649 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:21:51.669 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:21:51.671 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:21:51.672 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:21:51.680 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:21:51.685 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.685 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:51.687 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:51.702 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:21:51.702 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:51.704 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:51.705 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:51.709 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:51.709 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.710 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:21:51.712 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:21:51.713 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:21:51.712330+08:00, lastProcessTime=2025-10-27T11:21:41.717027+08:00 2025-10-27 11:21:51.720 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:21:51.725 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:21:51.726 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:21:51.739 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:21:51.741 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:21:51.755 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:21:51.756 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:21:51.762 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:21:51.767 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:51.768 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:21:51.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:51.769 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.769 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:21:51.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.769 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:51.770 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:51.771 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:21:51.775 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:51.784 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:21:51.785 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:51.786 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:51.794 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:51.802 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:21:51.802 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:21:51.805 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:21:51.815 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:21:51.819 INFO 2458 --- [host-check-host-status4,8a7d0020c497a07c,d4e295ff5bf796d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:21:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:51.836 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:21:51.838 INFO 2458 --- [host-check-host-status3,8a7d0020c497a07c,beb24f0db26e60ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:21:51.841 INFO 2458 --- [host-check-host-status5,8a7d0020c497a07c,42c60b1b22a6974f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:21:51.844 INFO 2458 --- [host-check-host-status2,8a7d0020c497a07c,7b3489af11c1e66d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:21:51.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:51.855 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.856 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:51.857 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:51.857 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:21:51.858 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.858 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:21:51.860 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:51.863 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:21:51.864 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:51.866 INFO 2458 --- [host-check-host-status1,8a7d0020c497a07c,70cc7b28fdfd201c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:21:51.870 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:21:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:21:51.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:21:51.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:51.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:51.958 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:51.958 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:51.959 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:21:51.989 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:21:51.712330+08:00 2025-10-27 11:21:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:52.001 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:21:52.001 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.002 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:21:52.004 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.005 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:21:52.020 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:21:52.028 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:52.028 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:21:52.042 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:52.043 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:21:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:21:52.066 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:21:52.066 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:21:52.104 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:21:52.140 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:21:52.152 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:21:52.162 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:21:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:21:52.180 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:21:52.239 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:21:52.302 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:21:52.303 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.304 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:21:52.305 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.306 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:21:52.342 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:21:52.382 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:21:52.382 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:21:52.383 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:21:52.420 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:21:52.420 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:21:52.420 INFO 2458 --- [ocp-schedules-1,1e4f3e04206f948e,a6f4c453622c07b9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:21:52.565 WARN 2458 --- [ocp-async-7,deb87cd38d7cd9dd,894a97f36055b05c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:52.565 INFO 2458 --- [ocp-async-6,9f152dbc4f9b9fa2,5b1f349bf18984a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:53.164 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:21:53.233 INFO 2458 --- [host-check-clockdiff3,e9feeeb2f9d5b6ba,c902c53365f6c023] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535313 0 0 , error= 2025-10-27 11:21:53.233 INFO 2458 --- [ocp-schedules-6,6309d5b6990fad87,70cfb0d2b8da96d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20396,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5096,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:21:53.234 INFO 2458 --- [ocp-schedules-6,6309d5b6990fad87,70cfb0d2b8da96d7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:21:53.707 INFO 2458 --- [prometheus-minute-schedule-1,16b6e08106e968aa,5245a629793159f7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:53.715 INFO 2458 --- [prometheus-minute-schedule-1,16b6e08106e968aa,5245a629793159f7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:54.295 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:54.297 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:54.565 INFO 2458 --- [ocp-async-9,ffe69f4e8c2d38e8,404239dc8e1fb82a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:54.565 WARN 2458 --- [ocp-async-10,9fc013e926b2bab6,3896cc7d1933f10b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:54.665 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,561f3d3b6808f6a0,bbe880287dae4343] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=561f3d3b6808f6a0, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:21:54.665 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,919bc3e25bdca298,1fe896297149024d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=919bc3e25bdca298, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:21:54.666 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,d8a75a6db39cefb1,167ee0e9b5a49647] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=d8a75a6db39cefb1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:21:54.666 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,d8a75a6db39cefb1,167ee0e9b5a49647] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=d8a75a6db39cefb1, duration=17 ms] 2025-10-27 11:21:54.666 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,919bc3e25bdca298,1fe896297149024d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=919bc3e25bdca298, duration=17 ms] 2025-10-27 11:21:54.674 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,bf6a7e8e4cc86369,3bb42185e9f117fc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/metricGroups?scope=TENANT&type=TOP, client=10.40.198.13, traceId=bf6a7e8e4cc86369, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.MonitorController.listMetricClasses(Pageable,MetricType,MetricScope,VersionTarget,Long), args=Page request [number: 0, size 5000, sort: id: ASC],TOP,TENANT,OBCLUSTER,] 2025-10-27 11:21:54.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,6965e6e4daf6a8e4,8007a22bc84e907f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=6965e6e4daf6a8e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:21:54.680 INFO 2458 --- [ocp-schedules-17,08443e4f38a42746,8092811f99a48a31] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:21:54.683 INFO 2458 --- [ocp-schedules-17,08443e4f38a42746,8092811f99a48a31] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:21:54.684 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,561f3d3b6808f6a0,bbe880287dae4343] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=561f3d3b6808f6a0, duration=35 ms] 2025-10-27 11:21:54.699 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,bf6a7e8e4cc86369,3bb42185e9f117fc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/metricGroups client=10.40.198.13, traceId=bf6a7e8e4cc86369, duration=58 ms] 2025-10-27 11:21:54.762 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,57baec9d292b1c3c,ba0e63efb6dd8500] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenantsView?page=1&size=10, client=10.40.198.13, traceId=57baec9d292b1c3c, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenantsView(Pageable,String,List,Boolean,String,Boolean,List,List), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:21:54.794 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,faf7af2d1bd3c17b,f9c06aa1ddab2020] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=faf7af2d1bd3c17b, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:21:54.804 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,57baec9d292b1c3c,ba0e63efb6dd8500] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenantsView client=10.40.198.13, traceId=57baec9d292b1c3c, duration=52 ms] 2025-10-27 11:21:54.834 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,faf7af2d1bd3c17b,f9c06aa1ddab2020] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=faf7af2d1bd3c17b, duration=48 ms] 2025-10-27 11:21:54.858 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:21:54.858 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:21:54.858 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:21:54.903 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,6965e6e4daf6a8e4,8007a22bc84e907f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=6965e6e4daf6a8e4, duration=254 ms] 2025-10-27 11:21:55.092 INFO 2458 --- [ocp-schedules-21,39f84c55999fca4e,91c8d7d614a31eca] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:21:55.332 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,905db53ab0de9f21,de71aafa4feacdbe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A21%3A55%2B08%3A00&endTime=2025-10-27T11%3A21%3A55%2B08%3A00&metrics=sql_all_count&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=905db53ab0de9f21, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:21:55+08:00,2025-10-27T11:21:55+08:00,sql_all_count,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:21:55.332 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,374217239207fcf8,c2c21a877369dd95] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A21%3A55%2B08%3A00&endTime=2025-10-27T11%3A21%3A55%2B08%3A00&metrics=sql_all_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=374217239207fcf8, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:21:55+08:00,2025-10-27T11:21:55+08:00,sql_all_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:21:55.332 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,efc9332163172f99,5ff82f24e3020835] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A21%3A55%2B08%3A00&endTime=2025-10-27T11%3A21%3A55%2B08%3A00&metrics=tps&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=efc9332163172f99, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:21:55+08:00,2025-10-27T11:21:55+08:00,tps,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:21:55.342 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,37fe45cda386f784,e98f5a9b27cd9031] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/monitor/top?startTime=2025-10-27T10%3A21%3A55%2B08%3A00&endTime=2025-10-27T11%3A21%3A55%2B08%3A00&metrics=tps_rt&queryAsync=true&labels=app%3AOB&groupBy=app%2Cob_cluster_name%2Cob_cluster_id%2Ctenant_name&limit=5&maxPoints=360, client=10.40.198.13, traceId=37fe45cda386f784, method=IterableResponse com.oceanbase.ocp.server.common.controller.MonitorController.queryMetricTop(OffsetDateTime,OffsetDateTime,String,String,Long,Long,Integer,Integer,String,Boolean), args=2025-10-27T10:21:55+08:00,2025-10-27T11:21:55+08:00,tps_rt,app,ob_cluster_name,ob_cluster_id,tenant_name,0,0,5,360,app:OB,true,] 2025-10-27 11:21:55.417 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,efc9332163172f99,5ff82f24e3020835] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=efc9332163172f99, duration=93 ms] 2025-10-27 11:21:55.433 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,37fe45cda386f784,e98f5a9b27cd9031] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=37fe45cda386f784, duration=98 ms] 2025-10-27 11:21:55.447 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,905db53ab0de9f21,de71aafa4feacdbe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=905db53ab0de9f21, duration=123 ms] 2025-10-27 11:21:55.452 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,374217239207fcf8,c2c21a877369dd95] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/monitor/top client=10.40.198.13, traceId=374217239207fcf8, duration=128 ms] 2025-10-27 11:21:56.000 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,dd2f35851908e318,c2300e92d4307db1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clustersView?page=1&size=10, client=10.40.198.13, traceId=dd2f35851908e318, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClustersView(String,Pageable), args=Page request [number: 0, size 10, sort: name: ASC],] 2025-10-27 11:21:56.018 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,dd2f35851908e318,c2300e92d4307db1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clustersView client=10.40.198.13, traceId=dd2f35851908e318, duration=27 ms] 2025-10-27 11:21:56.022 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:21:56.022 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:21:56.538 INFO 2458 --- [ocp-schedules-29,9b798292b4f81489,aac2db43c3fa0076] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:21:56.565 WARN 2458 --- [ocp-async-4,f7c48e3d18a2b32b,6ce442e445927ecc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:56.565 INFO 2458 --- [ocp-async-3,33fd038d44bf4c0a,7b23427e8c8d5ee1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:56.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,6421fc0deb83cccd,4e9e8bb1c21fd7cf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=6421fc0deb83cccd, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:21:56.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,6421fc0deb83cccd,4e9e8bb1c21fd7cf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=6421fc0deb83cccd, duration=10 ms] 2025-10-27 11:21:56.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,be08fa4b7e667903,0c1d09a5c7d11a15] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=be08fa4b7e667903, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:21:56.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,be08fa4b7e667903,0c1d09a5c7d11a15] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=be08fa4b7e667903, duration=10 ms] 2025-10-27 11:21:56.605 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,93e16f7e09833977,9e3a59e8af817759] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=93e16f7e09833977, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:21:56.605 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,93e16f7e09833977,9e3a59e8af817759] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=93e16f7e09833977, duration=11 ms] 2025-10-27 11:21:56.707 INFO 2458 --- [prometheus-minute-schedule-1,c41e8b3c8593330b,afe4948fa4ccc7e9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:56.714 INFO 2458 --- [prometheus-minute-schedule-1,c41e8b3c8593330b,afe4948fa4ccc7e9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:56.787 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:21:56.787 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:21:56.787 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:21:56.984 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,997ed6adc3dea6ad,ef53963121bc284a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=997ed6adc3dea6ad, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:21:56.993 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,997ed6adc3dea6ad,ef53963121bc284a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=997ed6adc3dea6ad, duration=17 ms] 2025-10-27 11:21:57.297 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:21:57.298 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:21:58.077 INFO 2458 --- [ocp-schedules-1,dc1020cd9c80c0b1,e6c5822c02233840] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:21:58.086 INFO 2458 --- [ocp-schedules-1,dc1020cd9c80c0b1,e6c5822c02233840] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:21:58.139 INFO 2458 --- [ocp-schedules-1,dc1020cd9c80c0b1,e6c5822c02233840] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:21:58.523 INFO 2458 --- [ocp-async-3,8c20d9da1278ff24,0cb0e0cdce195372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:21:58.565 WARN 2458 --- [ocp-async-2,a150c7c27fb631f1,705a973c0c25d223] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:21:58.565 INFO 2458 --- [ocp-async-7,4efa0d130ce891ed,a54403df5afc9b0e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:21:59.375 INFO 2458 --- [ocp-schedules-3,3f6b593d6819e88f,8c77081273cccda2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:21:59.598 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,1fb20b12fbe72248,c8e7470f87df11e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=1fb20b12fbe72248, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:21:59.600 INFO 2458 --- [ocp-schedules-22,6235bb4457e6a618,4e8bebd0c9ec8dad] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:21:59.602 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,1fb20b12fbe72248,c8e7470f87df11e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=1fb20b12fbe72248, duration=11 ms] 2025-10-27 11:21:59.604 INFO 2458 --- [ocp-schedules-22,6235bb4457e6a618,4e8bebd0c9ec8dad] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:21:59.707 INFO 2458 --- [prometheus-minute-schedule-1,f44760752e75d2d7,20f817aa4099d372] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535320, interval=60 2025-10-27 11:21:59.717 INFO 2458 --- [prometheus-minute-schedule-1,f44760752e75d2d7,20f817aa4099d372] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:21:59.798 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e9e73b065534b919,cbf4fb0e96665547] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?type=BINLOG_SERVICE_INSTALL_PACKAGE&withInstallationData=true, client=10.40.198.13, traceId=e9e73b065534b919, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=[BINLOG_SERVICE_INSTALL_PACKAGE],true,Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:21:59.839 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,e9e73b065534b919,cbf4fb0e96665547] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.40.198.13, traceId=e9e73b065534b919, duration=50 ms] 2025-10-27 11:22:00.074 INFO 2458 --- [ocp-schedules-7,5288c29da7b26e30,05a814806e72075d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:00.098 WARN 2458 --- [scanner-112175532-1,5288c29da7b26e30,95ed97a4cb367e81] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:00.277 INFO 2458 --- [alarm-detect-15,5288c29da7b26e30,71df3268bbd78fa9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535290 ~ 1761535310 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535305, value=39.768)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535305, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535305=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:22:00.298 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:00.300 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:00.565 INFO 2458 --- [ocp-async-7,3a417c3c3b79cb7e,ce9e83da2f1b9e71] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:00.565 WARN 2458 --- [ocp-async-5,f30d3a178f4e86a2,fe6b92def0cc224b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:01.713857+08:00, lastProcessTime=2025-10-27T11:21:51.712330+08:00 2025-10-27 11:22:01.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:01.713857+08:00 2025-10-27 11:22:02.566 INFO 2458 --- [ocp-async-7,a0ca241d2307cb9b,ef141aad98c0d889] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:02.566 WARN 2458 --- [ocp-async-8,7c0c03827f9055bd,d018e87dccdd9e94] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:02.707 INFO 2458 --- [prometheus-minute-schedule-1,959485f1d63d4986,0d487f75d12ced1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:02.716 INFO 2458 --- [prometheus-minute-schedule-1,959485f1d63d4986,0d487f75d12ced1a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:03.300 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:03.301 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:03.615 INFO 2458 --- [ocp-schedules-14,a3976a981c9ca6be,1081dc40130619d6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:22:03.621 INFO 2458 --- [ocp-schedules-14,a3976a981c9ca6be,1081dc40130619d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:03.660 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:03.660 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:22:03.675 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:03.675 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:22:03.696 INFO 2458 --- [ocp-schedules-14,a3976a981c9ca6be,1081dc40130619d6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:22:04.566 INFO 2458 --- [ocp-async-2,6d79393df3fa8623,59ba1d788fac8686] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:04.566 WARN 2458 --- [ocp-async-6,81868da77e782e03,185e9e06eb5ab11f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:04.692 INFO 2458 --- [ocp-schedules-22,3fdc7e80438420cc,4cdb1d7fcc712358] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:04.695 INFO 2458 --- [ocp-schedules-22,3fdc7e80438420cc,4cdb1d7fcc712358] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:05.707 INFO 2458 --- [prometheus-minute-schedule-1,d65829947666b070,0dec6f8d54facc4c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:05.715 INFO 2458 --- [prometheus-minute-schedule-1,d65829947666b070,0dec6f8d54facc4c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:06.025 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:22:06.027 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:06.028 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:22:06.029 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:22:06.082 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:22:06.123 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:22:06.146 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:22:06.146 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:22:06.146 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:22:06.146 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-27T11:19:30, target=2025-10-27T11:17:30, granule=120000ms, delay=120000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:17:30, target=2025-10-27T11:12:30, granule=600000ms, delay=300000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:22:06.151 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=122ms, period=120000ms 2025-10-27 11:22:06.301 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:06.303 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:06.547 INFO 2458 --- [ocp-schedules-19,c5bb19293653a82e,2ec4f2179f62b9b7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:06.566 INFO 2458 --- [ocp-async-2,b08cb487f483ee67,76dcbf2c064a178c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:06.566 WARN 2458 --- [ocp-async-9,a9885a14fac93d65,25473dd7fa847f4c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:06.893 INFO 2458 --- [ocp-schedules-9,9a504194b504b269,74223e61454d87a0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:22:06.901 INFO 2458 --- [ocp-schedules-9,9a504194b504b269,74223e61454d87a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:06.995 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:22:06.995 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:06.998 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:22:07.007 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.007 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.033 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.051 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:22:07.051 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.055 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:22:07.061 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.062 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.064 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.064 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.074 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.075 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.093 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.093 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.101 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.102 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.123 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.124 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.130 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.131 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:22:07.133 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.134 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.152 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.153 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.153 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.154 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.155 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.155 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.156 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.157 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.157 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.158 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.159 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.160 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.160 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:22:07.162 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.162 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.163 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:22:07.191 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.192 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:22:07.194 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.194 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:22:07.209 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:07.210 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.210 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:22:07.215 INFO 2458 --- [ocp-schedules-10,2a48cbd803ca7a3a,0c4464d1d00cede7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:22:07.220 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.220 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:22:07.220 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.220 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:22:07.223 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:22:07.223 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:22:07.227 INFO 2458 --- [ocp-schedules-10,2a48cbd803ca7a3a,0c4464d1d00cede7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:07.227 INFO 2458 --- [ocp-schedules-10,2a48cbd803ca7a3a,0c4464d1d00cede7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:07.228 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.228 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:22:07.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:22:07.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:22:07.287 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.287 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:22:07.292 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9f00789e89b621b8,bfe00fe75f0ed762] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=9f00789e89b621b8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 11:22:07.293 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,dd4aac77eb4ff8fb,c1009e59da92fc8a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=dd4aac77eb4ff8fb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:22:07.294 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,fe8e717f34f8f941,3da63e9b85d42e71] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=fe8e717f34f8f941, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:22:07.294 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,fe8e717f34f8f941,3da63e9b85d42e71] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=fe8e717f34f8f941, duration=11 ms] 2025-10-27 11:22:07.296 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,97c059cd74c81473,e6b94cf3f1208b2f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters?null, client=10.40.198.13, traceId=97c059cd74c81473, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClusters(Pageable), args=Page request [number: 0, size 5000, sort: name: ASC],] 2025-10-27 11:22:07.297 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,c80361c5c104d5e4,cb4bd0d5868a5405] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=c80361c5c104d5e4, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:22:07.322 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,dd4aac77eb4ff8fb,c1009e59da92fc8a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=dd4aac77eb4ff8fb, duration=39 ms] 2025-10-27 11:22:07.336 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:22:07.340 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9f00789e89b621b8,bfe00fe75f0ed762] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:22:07.340 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,97c059cd74c81473,e6b94cf3f1208b2f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters client=10.40.198.13, traceId=97c059cd74c81473, duration=57 ms] 2025-10-27 11:22:07.344 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9f00789e89b621b8,bfe00fe75f0ed762] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:22:07.349 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,c80361c5c104d5e4,cb4bd0d5868a5405] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=c80361c5c104d5e4, duration=66 ms] 2025-10-27 11:22:07.362 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,d2df057bd3fe5f5d,f88ade083fa4db86] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=d2df057bd3fe5f5d, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:22:07.364 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,d2df057bd3fe5f5d,f88ade083fa4db86] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=d2df057bd3fe5f5d, duration=9 ms] 2025-10-27 11:22:07.364 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,d9b570e7a3312304,aa2289e60ee64fff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=d9b570e7a3312304, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:22:07.380 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6d4c3e7c0643f19e,9d0736f5de99a3d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=6d4c3e7c0643f19e, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:22:07.395 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.396 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.397 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.397 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.398 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,6d4c3e7c0643f19e,9d0736f5de99a3d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=6d4c3e7c0643f19e, duration=26 ms] 2025-10-27 11:22:07.399 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.399 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.400 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.401 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.401 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.402 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.403 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.404 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.405 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.406 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.407 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.408 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.409 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.410 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.411 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.411 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:22:07.412 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.413 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:22:07.414 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.415 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:22:07.415 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:22:07.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:22:07.471 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9f00789e89b621b8,bfe00fe75f0ed762] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=9f00789e89b621b8, duration=188 ms] 2025-10-27 11:22:07.474 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:22:07.474 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:22:07.502 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,27a627d4e97b562a,5dc6a41730f7d7f6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=27a627d4e97b562a, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 11:22:07.507 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:22:07.507 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:22:07.507 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:22:07.518 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:07.518 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:07.519 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:22:07.527 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,27a627d4e97b562a,5dc6a41730f7d7f6] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:22:07.527 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,27a627d4e97b562a,5dc6a41730f7d7f6] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:22:07.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:22:07.549 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:22:07.549 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:22:07.549 INFO 2458 --- [ocp-schedules-9,9a504194b504b269,74223e61454d87a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:22:07.599 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,d9b570e7a3312304,aa2289e60ee64fff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=d9b570e7a3312304, duration=244 ms] 2025-10-27 11:22:07.654 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,27a627d4e97b562a,5dc6a41730f7d7f6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=27a627d4e97b562a, duration=162 ms] 2025-10-27 11:22:08.523 INFO 2458 --- [ocp-async-2,da70b4fe62ea7750,5c595d812f809553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:08.566 INFO 2458 --- [ocp-async-4,4a8f4face259576a,13c7c54c1c1c4ace] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:08.566 WARN 2458 --- [ocp-async-3,4c811b093fc55470,fcfae51707f24f20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:08.707 INFO 2458 --- [prometheus-minute-schedule-1,32e481086d624558,b0fbe532ed2db802] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:08.715 INFO 2458 --- [prometheus-minute-schedule-1,32e481086d624558,b0fbe532ed2db802] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:09.284 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,73411e17ad502eb7,de9c49728ee2d815] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=73411e17ad502eb7, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:22:09.284 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,fbf611c2905736eb,6ce3cdf79d2bf5d3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=fbf611c2905736eb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:22:09.284 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,73411e17ad502eb7,de9c49728ee2d815] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=73411e17ad502eb7, duration=9 ms] 2025-10-27 11:22:09.284 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,fbf611c2905736eb,6ce3cdf79d2bf5d3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=fbf611c2905736eb, duration=8 ms] 2025-10-27 11:22:09.285 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,d5cd1389f3b4d088,4b64e257660ec6b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=d5cd1389f3b4d088, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:22:09.285 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,d5cd1389f3b4d088,4b64e257660ec6b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=d5cd1389f3b4d088, duration=9 ms] 2025-10-27 11:22:09.303 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:09.304 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:09.304 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:22:09.304 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:22:09.304 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:22:10.022 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:22:10.022 INFO 2458 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:22:10.060 INFO 2458 --- [Check obproxy1,5b175e11d2c2d3f6,f6e6a5bf23a2218a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:22:10.060 INFO 2458 --- [Collect obproxy1,36f26b26ce093a98,1892c85760fdf735] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:22:10.060 INFO 2458 --- [Check obproxy1,5b175e11d2c2d3f6,f6e6a5bf23a2218a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:22:10.060 INFO 2458 --- [Collect obproxy1,36f26b26ce093a98,1892c85760fdf735] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:22:10.091 INFO 2458 --- [Collect obproxy1,36f26b26ce093a98,1892c85760fdf735] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:22:10.092 INFO 2458 --- [Check obproxy1,5b175e11d2c2d3f6,f6e6a5bf23a2218a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:22:10.094 INFO 2458 --- [Collect obproxy1,36f26b26ce093a98,1892c85760fdf735] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:22:10.095 INFO 2458 --- [Check obproxy1,5b175e11d2c2d3f6,f6e6a5bf23a2218a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:22:10.095 INFO 2458 --- [Check obproxy1,5b175e11d2c2d3f6,f6e6a5bf23a2218a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:22:10.097 INFO 2458 --- [Collect obproxy1,36f26b26ce093a98,1892c85760fdf735] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:22:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:22:10.286 INFO 2458 --- [ocp-schedules-19,2aca9fe3baacde4a,cfe1bf49ad898dde] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:10.297 WARN 2458 --- [scanner-112175690-1,2aca9fe3baacde4a,8fd7dc6d463c5c24] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:10.339 INFO 2458 --- [alarm-detect-2,2aca9fe3baacde4a,0c7eb03405deb064] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:10.343 INFO 2458 --- [alarm-detect-2,2aca9fe3baacde4a,0c7eb03405deb064] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:10.352 INFO 2458 --- [alarm-detect-2,2aca9fe3baacde4a,9b1c6e4e2e7432d0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:10.353 INFO 2458 --- [alarm-detect-18,2aca9fe3baacde4a,6b67d0abbc1c7121] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:10.356 INFO 2458 --- [alarm-detect-2,2aca9fe3baacde4a,9b1c6e4e2e7432d0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:10.358 INFO 2458 --- [alarm-detect-18,2aca9fe3baacde4a,6b67d0abbc1c7121] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:10.438 INFO 2458 --- [alarm-detect-1,2aca9fe3baacde4a,0cff91eeb8b53e3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:10.443 INFO 2458 --- [alarm-detect-1,2aca9fe3baacde4a,0cff91eeb8b53e3b] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:10.469 INFO 2458 --- [alarm-detect-3,2aca9fe3baacde4a,d47775e81648e996] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535305 ~ 1761535320 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535320, value=39.352)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535320, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535320=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:22:10.566 INFO 2458 --- [ocp-async-4,3cb6f0669ce84aa4,6ed948ab75cae2be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:10.566 WARN 2458 --- [ocp-async-5,833d62a2b67badc8,f416e92ad0586bd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:10.836 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,920f304dceb2e491,2dac61d96e7388b2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=920f304dceb2e491, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:22:10.836 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,920f304dceb2e491,2dac61d96e7388b2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=920f304dceb2e491, duration=3 ms] 2025-10-27 11:22:11.707 INFO 2458 --- [prometheus-minute-schedule-1,0c980d12a47f4d71,a915104b38faa0fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:11.714166+08:00, lastProcessTime=2025-10-27T11:22:01.713857+08:00 2025-10-27 11:22:11.715 INFO 2458 --- [prometheus-minute-schedule-1,0c980d12a47f4d71,a915104b38faa0fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:11.938 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:11.714166+08:00 2025-10-27 11:22:12.276 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,09dc1af88589fc30,2f2394f68ddbd2f4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=09dc1af88589fc30, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:22:12.280 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,09dc1af88589fc30,2f2394f68ddbd2f4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=09dc1af88589fc30, duration=15 ms] 2025-10-27 11:22:12.304 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:12.304 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:12.309 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=1000, affectRows=1000 2025-10-27 11:22:12.313 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=994, validCount=992 2025-10-27 11:22:12.505 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134601996) Duplicate entry '835014129065861120-1761535200' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134601996) Duplicate entry '835014129065861120-1761535200' for key 'PRIMARY' 2025-10-27 11:22:12.566 INFO 2458 --- [ocp-async-4,b351ba7c8178a12f,8fbc89b5626a8963] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:12.566 WARN 2458 --- [ocp-async-8,4e1c28bd63682af9,9bc64cfe54a29452] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:12.795 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535260, nodeCount=817, affectRows=817 2025-10-27 11:22:12.840 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=1000, affectRows=1000 2025-10-27 11:22:12.844 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=999, validCount=995 2025-10-27 11:22:12.860 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134521640) Duplicate entry '835370603008958464-1761535200' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134521640) Duplicate entry '835370603008958464-1761535200' for key 'PRIMARY' 2025-10-27 11:22:13.733 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=508, affectRows=508 2025-10-27 11:22:13.735 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=507, validCount=505 2025-10-27 11:22:13.769 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=588, affectRows=588 2025-10-27 11:22:13.772 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=588, validCount=586 2025-10-27 11:22:14.566 INFO 2458 --- [ocp-async-3,82fb56f95e78e1cf,b04370df13fec163] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:14.566 WARN 2458 --- [ocp-async-7,76252d5cb9494b70,616bf442977cb6f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:14.702 INFO 2458 --- [ocp-schedules-11,8000a779b34e4892,a535cc73bfb2d8e9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:14.706 INFO 2458 --- [ocp-schedules-11,8000a779b34e4892,a535cc73bfb2d8e9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:14.707 INFO 2458 --- [prometheus-minute-schedule-1,2e9cbf68cf9f8bc5,48edd53cce19faa5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:14.714 INFO 2458 --- [prometheus-minute-schedule-1,2e9cbf68cf9f8bc5,48edd53cce19faa5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:15.305 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:15.305 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:15.351 INFO 2458 --- [ocp-schedules-8,6cebf25ba9ac3420,6693459b672acc56] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:22:15.351+08:00, clusters=[1, 2] 2025-10-27 11:22:15.354 INFO 2458 --- [ocp-schedules-8,6cebf25ba9ac3420,6693459b672acc56] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:22:15.355 INFO 2458 --- [ocp-schedules-8,6cebf25ba9ac3420,6693459b672acc56] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:22:16.511 INFO 2458 --- [ocp-schedules-30,6c6ac72229e0288a,4bfedc373dca9646] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:22:16.511 INFO 2458 --- [ocp-schedules-30,6c6ac72229e0288a,4bfedc373dca9646] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:22:16.554 INFO 2458 --- [ocp-schedules-29,f9405a9bc1b05ab4,9eab26ce2c2a6d14] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:16.567 INFO 2458 --- [ocp-async-5,ad998d8f5285754c,6bed386a0072aa3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:16.567 WARN 2458 --- [ocp-async-1,fe04db4194c45d1b,33387e127c0cd1a0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:16.569 INFO 2458 --- [Collect all tenants major compactions1,6c6ac72229e0288a,e726850c2e5bc54d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:22:16.570 INFO 2458 --- [Collect all tenants major compactions2,6c6ac72229e0288a,44ae2e9f7b8cc13a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:22:16.570 INFO 2458 --- [Collect all tenants major compactions4,6c6ac72229e0288a,2b283531a7beaece] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:22:16.570 INFO 2458 --- [Collect all tenants major compactions3,6c6ac72229e0288a,9eda2328e33d555d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:22:16.571 INFO 2458 --- [Collect all tenants major compactions5,6c6ac72229e0288a,fdcd9790565a51e3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:22:16.571 INFO 2458 --- [Collect all tenants major compactions6,6c6ac72229e0288a,195edeaeda8f8c11] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:22:16.626 INFO 2458 --- [Collect all tenants major compactions6,6c6ac72229e0288a,195edeaeda8f8c11] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.626 INFO 2458 --- [Collect all tenants major compactions5,6c6ac72229e0288a,fdcd9790565a51e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.627 INFO 2458 --- [Collect all tenants major compactions6,6c6ac72229e0288a,195edeaeda8f8c11] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:22:16.628 INFO 2458 --- [Collect all tenants major compactions5,6c6ac72229e0288a,fdcd9790565a51e3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:22:16.633 INFO 2458 --- [Collect all tenants major compactions6,6c6ac72229e0288a,195edeaeda8f8c11] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:22:16.635 INFO 2458 --- [Collect all tenants major compactions5,6c6ac72229e0288a,fdcd9790565a51e3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:22:16.653 INFO 2458 --- [Collect all tenants major compactions1,6c6ac72229e0288a,e726850c2e5bc54d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.655 INFO 2458 --- [Collect all tenants major compactions1,6c6ac72229e0288a,e726850c2e5bc54d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:22:16.661 INFO 2458 --- [Collect all tenants major compactions1,6c6ac72229e0288a,e726850c2e5bc54d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:22:16.665 INFO 2458 --- [Collect all tenants major compactions3,6c6ac72229e0288a,9eda2328e33d555d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.667 INFO 2458 --- [Collect all tenants major compactions3,6c6ac72229e0288a,9eda2328e33d555d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:22:16.667 INFO 2458 --- [Collect all tenants major compactions2,6c6ac72229e0288a,44ae2e9f7b8cc13a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.668 INFO 2458 --- [Collect all tenants major compactions2,6c6ac72229e0288a,44ae2e9f7b8cc13a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:22:16.670 INFO 2458 --- [Collect all tenants major compactions3,6c6ac72229e0288a,9eda2328e33d555d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:22:16.672 INFO 2458 --- [Collect all tenants major compactions2,6c6ac72229e0288a,44ae2e9f7b8cc13a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:22:16.683 INFO 2458 --- [Collect all tenants major compactions4,6c6ac72229e0288a,2b283531a7beaece] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:16.685 INFO 2458 --- [Collect all tenants major compactions4,6c6ac72229e0288a,2b283531a7beaece] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:22:16.689 INFO 2458 --- [Collect all tenants major compactions4,6c6ac72229e0288a,2b283531a7beaece] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:22:16.691 INFO 2458 --- [ocp-schedules-30,6c6ac72229e0288a,4bfedc373dca9646] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:22:17.707 INFO 2458 --- [prometheus-minute-schedule-1,25273fdcfb5ba20b,0ec48745e445398b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:17.715 INFO 2458 --- [prometheus-minute-schedule-1,25273fdcfb5ba20b,0ec48745e445398b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:18.305 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:18.307 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:18.523 INFO 2458 --- [ocp-async-5,7818169a12a5b7e0,d9d80c397a0ae7c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:18.567 INFO 2458 --- [ocp-async-10,6f6ea782ac73b928,94456a507126c666] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:18.567 WARN 2458 --- [ocp-async-4,ff8777183b894b3f,29e76ab7dae9409e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:18.712 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708} 2025-10-27 11:22:18.718 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicS 2025-10-27 11:22:18.719 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:18.729 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:18.729 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:22:18.729 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:18.908 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761448938908000, endTimeUs=1761535218708000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:22:18.908Z, endTime=2025-10-27T03:20:18.708Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:22:18.940 INFO 2458 --- [ocp-schedules-18,ecfcfe84c1e6fb63,d220dd70ba8c1209] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:18.941 INFO 2458 --- [ocp-schedules-18,ecfcfe84c1e6fb63,d220dd70ba8c1209] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:22:19.691 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:22:19.691 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:22:19.732 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:22:19.732 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535218708000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:22:19.746 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:22:19.746 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:22:19.782 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:22:19.782 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535218708000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:22:19.786 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:22:19.786 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:22:18.908, end=2025-10-27T11:20:18.708}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:22:19.786 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:19.787 INFO 2458 --- [ocp-schedules-22,d03b46e7cfdbc2b2,abc9901db30d7d4b] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:22:19.787 INFO 2458 --- [ocp-schedules-22,d03b46e7cfdbc2b2,abc9901db30d7d4b] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:22:19.788 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:19.788 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:22:19.788 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:19.790 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:19.795 INFO 2458 --- [ocp-schedules-22,d03b46e7cfdbc2b2,abc9901db30d7d4b] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:22:19.795 INFO 2458 --- [ocp-schedules-22,d03b46e7cfdbc2b2,abc9901db30d7d4b] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:22:19.796 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:19.796 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:22:19.796 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:19.803 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:19.805 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:19.805 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:22:19.805 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:19.807 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:19.809 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:19.809 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:22:19.809 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:19.812 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:22:19.814 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:18:17.928, end=2025-10-27T11:20:18.708},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:22:19.814 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:22:19.814 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:22:19.816 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:22:20.022 INFO 2458 --- [ocp-schedules-10,efa95e81adc3de11,a491c5d23e3ffbba] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:22:20.474 INFO 2458 --- [ocp-schedules-18,2c7360ba0e021c10,a67878eac429a907] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:20.489 WARN 2458 --- [scanner-112177471-1,2c7360ba0e021c10,39b60f8d37084f79] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:20.567 WARN 2458 --- [ocp-async-2,d7a1c7909d16db2a,e330e9978cbb339e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:20.567 INFO 2458 --- [ocp-async-3,7d6aca205533c92d,7454a30d3b4c4ab1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:20.707 INFO 2458 --- [prometheus-minute-schedule-1,117a3c01f5b3abe1,8cb0be3b74190094] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:20.719 INFO 2458 --- [prometheus-minute-schedule-1,117a3c01f5b3abe1,8cb0be3b74190094] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:21.307 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:21.308 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:21.409 INFO 2458 --- [ocp-schedules-9,200af0c7c089e0d3,947658547fa9867d] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:21.410 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:22:21.410 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:22:21.410 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:22:21.410 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:22:21.410 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:22:21.416 INFO 2458 --- [ocp-schedules-6,f515d7ea0bedf600,6409b561b67a13e1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:21.417 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:22:21.418 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:22:21.418 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:22:21.418 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:22:21.418 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:22:21.425 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:21.426 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:21.426 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:21.427 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:21.429 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:21.429 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:21.430 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:21.431 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:21.431 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:21.431 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:21.432 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:21.432 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:21.432 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:21.433 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:21.433 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:21.434 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:21.434 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:21.435 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:21.435 INFO 2458 --- [ocp-schedules-20,d04df2f02874d049,fec042382d014ec4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:21.435 INFO 2458 --- [host-check-clockdiff1,d04df2f02874d049,3f328b03cb4c7e65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:22:21.436 INFO 2458 --- [host-check-clockdiff2,d04df2f02874d049,d0edff452a4041f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:22:21.437 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:21.438 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:21.439 INFO 2458 --- [host-check-clockdiff3,d04df2f02874d049,06213966f653b3e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:22:21.439 INFO 2458 --- [host-check-clockdiff4,d04df2f02874d049,8fb987e9683a92e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:22:21.450 INFO 2458 --- [ocp-schedules-12,5a1106d27ec0d238,38e17af8f3e99726] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:22:21.450 INFO 2458 --- [host-check-clockdiff5,d04df2f02874d049,7fc942d9b5c91605] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:22:21.460 INFO 2458 --- [host-check-clockdiff4,d04df2f02874d049,8fb987e9683a92e0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535341 -1 -1 , error= 2025-10-27 11:22:21.461 INFO 2458 --- [host-check-clockdiff2,d04df2f02874d049,d0edff452a4041f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535341 -1 -1 , error= 2025-10-27 11:22:21.463 INFO 2458 --- [host-check-clockdiff3,d04df2f02874d049,06213966f653b3e4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535341 -1 -1 , error= 2025-10-27 11:22:21.465 INFO 2458 --- [host-check-clockdiff1,d04df2f02874d049,3f328b03cb4c7e65] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535341 0 0 , error= 2025-10-27 11:22:21.545 INFO 2458 --- [ocp-schedules-5,c5e297fb84600632,dc8a0a9ae03c8c09] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:22:21.549 INFO 2458 --- [ocp-schedules-15,68b1495520d07ce2,864428743f94192e] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:22:21.550 INFO 2458 --- [ocp-schedules-5,c5e297fb84600632,dc8a0a9ae03c8c09] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:21.554 INFO 2458 --- [ocp-schedules-26,f204df0738449975,d9ec68bfa9e5b317] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:22:21.555 INFO 2458 --- [ocp-schedules-15,68b1495520d07ce2,864428743f94192e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:21.557 INFO 2458 --- [ocp-schedules-15,68b1495520d07ce2,864428743f94192e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:22:21.561 INFO 2458 --- [ocp-schedules-26,f204df0738449975,d9ec68bfa9e5b317] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:21.568 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:22:21.570 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:22:21.570692547+08:00 2025-10-27 11:22:21.571 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:22:21.570+08:00 2025-10-27 11:22:21.576 INFO 2458 --- [ocp-schedules-11,aae21f1b3cf6b77b,b3e7dd4dc3644a9a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:22:21.581 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:22:21.570+08:00 and status AVAILABLE 2025-10-27 11:22:21.587 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:22:21.588 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:22:21.590 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:22:21.588784952+08:00 2025-10-27 11:22:21.593 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:22:21.590+08:00 2025-10-27 11:22:21.602 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:22:21.590+08:00 and status AVAILABLE 2025-10-27 11:22:21.603 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:22:21.608 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:22:21.608 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:22:21.608 INFO 2458 --- [host-check-agent-status3,200af0c7c089e0d3,8f83f02872544a37] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:22:21.608 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.608 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:22:21.609 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:21.610 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:22:21.615 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:22:21.616 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:22:21.618 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:22:21.618 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:22:21.618 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:22:21.618 INFO 2458 --- [host-check-agent-status5,200af0c7c089e0d3,5dd04a7ef1a7091b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:22:21.619 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:22:21.619 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:22:21.619 INFO 2458 --- [host-check-agent-status2,200af0c7c089e0d3,02fdd8a7dd429d64] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:22:21.619 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:22:21.622 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:22:21.622 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:22:21.622 INFO 2458 --- [host-check-agent-status4,200af0c7c089e0d3,49dd38c00bfda044] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:22:21.623 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:22:21.623 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:22:21.623 INFO 2458 --- [host-check-agent-status1,200af0c7c089e0d3,77d2dfe300803a2b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:22:21.624 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:21.627 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:21.628 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:22:21.629 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:22:21.629036518+08:00 2025-10-27 11:22:21.629 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:22:21.629+08:00 2025-10-27 11:22:21.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:21.630 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:22:21.631 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:22:21.630504649+08:00 2025-10-27 11:22:21.631 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:22:21.631+08:00 2025-10-27 11:22:21.633 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:22:21.634 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:22:21.633116889+08:00 2025-10-27 11:22:21.635 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:22:21.634+08:00 2025-10-27 11:22:21.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:22:21.637 INFO 2458 --- [ocp-schedules-5,c5e297fb84600632,dc8a0a9ae03c8c09] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:22:21.638 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:21.639 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:22:21.629+08:00 and status AVAILABLE 2025-10-27 11:22:21.639 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:22:21.631+08:00 and status AVAILABLE 2025-10-27 11:22:21.641 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:21.641 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.641 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:21.642 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:22:21.634+08:00 and status AVAILABLE 2025-10-27 11:22:21.643 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:22:21.645 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:22:21.648 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:22:21.667 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:21.669 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:21.670 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:21.671 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:21.672 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:21.675 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.691 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:21.695 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:21.710 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:21.711 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:21.711 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.712 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:22:21.712 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:22:21.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:21.719998+08:00, lastProcessTime=2025-10-27T11:22:11.714166+08:00 2025-10-27 11:22:21.734 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:22:21.744 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:22:21.748 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:21.750 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:21.750 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:21.771 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:22:21.772 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:22:21.778 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:22:21.779 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=1000, affectRows=1000 2025-10-27 11:22:21.781 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:22:21.783 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=1000, validCount=998 2025-10-27 11:22:21.795 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:21.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.796 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:21.797 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.798 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:21.798 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.799 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:22:21.799 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:21.800 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:22:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:21.804 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:22:21.806 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:22:21.815 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:22:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:21.820 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:22:21.823 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:21.843 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:22:21.861 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:21.862 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:21.862 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:21.871 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:22:21.871 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:21.881 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:22:21.883 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:22:21.885 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535260, nodeCount=1000, affectRows=1000 2025-10-27 11:22:21.889 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535260, nodeCount=992, validCount=978 2025-10-27 11:22:21.890 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:22:21.895 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:22:21.897 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:21.898 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.899 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:21.900 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:22:21.911 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:22:21.913 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:22:21.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:21.917 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:22:21.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:21.921 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:22:21.923 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:22:21.923 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:21.926 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:21.951 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:21.957 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:21.963 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:21.965 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:21.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:21.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:21.968 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:21.972 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:22:21.974 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:22:21.981 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:22:21.984 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:22:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:22:21.996 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:22:22.001 INFO 2458 --- [host-check-host-status2,f515d7ea0bedf600,77d2253cf1a8968e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:22:22.003 INFO 2458 --- [host-check-host-status5,f515d7ea0bedf600,48ba4fbbcf582c43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:22:22.009 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:22.009 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:22.010 INFO 2458 --- [host-check-host-status3,f515d7ea0bedf600,b2f6d8c3ba68048b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:22:22.010 INFO 2458 --- [host-check-host-status4,f515d7ea0bedf600,b933ab32ceac4f41] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:22:22.023 INFO 2458 --- [host-check-host-status1,f515d7ea0bedf600,15683d1bff3d3585] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:22:22.036 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:22:22.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:22:22.039 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.040 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:22:22.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:22.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:22.057 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:22:22.082 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:22.082 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.083 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:22:22.094 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:22.094 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:22.105 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:22:22.105 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:22:22.130 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:22.130 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:22.143 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:22.143 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2025-10-27 11:22:22.143 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2025-10-27 11:22:22.143 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2025-10-27 11:22:22.143 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2025-10-27 11:22:22.143 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2025-10-27 11:22:22.159 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.m.HostAgentConfigManagerImpl : binlogCluster BinlogCluster(id=4, name=binlogtest2, version=4.0.1-100000162024122511, status=RUNNING, metadbInfo=BinlogClusterMetadbInfo(tenantId=2, username=root, database=binlog_database, tenantName=ocp_meta, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187, clusterType=PRIMARY, obVersion=4.3.5.4, address=10.40.65.146, port=2881, fullUserName=root@ocp_meta), resourceStats=null, servers=[BinlogServer(id=4, ip=10.40.65.151, port=2983, binlogClusterId=4, binlogClusterName=binlogtest2, hostId=4, hostName=tjtx40-65-151.zhuanos.com, hostKind=DEDICATED_PHYSICAL_MACHINE, hostTypeName=test, idcName=default_idc, regionName=deault_region, status=RUNNING, resourceStats=null)], instances=null, tags=null, createTime=2025-10-24T17:01:22+08:00, updateTime=2025-10-24T17:04:41+08:00, installPath=/home/admin/binlogservice, binlogFilePath=/home/admin/binlogservice/run, address=10.40.65.151, port=2983) 2025-10-27 11:22:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:22.230 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:22.236 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.237 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.245 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:22:22.262 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:22:22.284 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:22:22.284 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:22:22.284 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:22:22.286 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:22:22.286 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:22:22.286 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:22:22.288 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2025-10-27 11:22:22.288 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:22.289 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:22:22.289 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:22:22.289 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:22:22.290 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2025-10-27 11:22:22.291 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:22:22.292 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2025-10-27 11:22:22.293 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2025-10-27 11:22:22.294 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:22.303 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:22.303 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2 to set sql collect property inactive 2025-10-27 11:22:22.304 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2025-10-27 11:22:22.306 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:22.306 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:22.307 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:22.310 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:22.311 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:22.313 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:22.315 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:22.317 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2025-10-27 11:22:22.328 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 3 to set sql collect property inactive 2025-10-27 11:22:22.331 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:22.337 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:22.359 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.binlog.cluster.id=4), (monagent.binlog.cluster.name=binlogtest2), (monagent.pipeline.logproxy.status=active), (binlog.multi.instance.config={"instanceConfigs":[],"instanceType":"binlog"}), (binlog.install.path=/home/admin/binlogservice), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_hardware_error","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["Hardware Error"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (mo 2025-10-27 11:22:22.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:22:22.399 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2025-10-27 11:22:22.404 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.146), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.146:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:22:22.404 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.168), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.168:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:22:22.422 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510188), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.66.107), (ob.run.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.66.107:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase2), (ob.logtailer.enabled=true), (ob.l 2025-10-27 11:22:22.424 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.67.22), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.67.22:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtailer. 2025-10-27 11:22:22.429 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2025-10-27 11:22:22.432 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2025-10-27 11:22:22.438 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2025-10-27 11:22:22.448 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2025-10-27 11:22:22.451 INFO 2458 --- [ocp-schedules-18,3e0985f1913cfb4f,de4df4c4df23c50f] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-10-27 11:22:22.458 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:22:22.459 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.459 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:22:22.461 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.462 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:22:22.482 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:21.719998+08:00 2025-10-27 11:22:22.497 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:22:22.536 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:22.537 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:22.537 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:22:22.567 WARN 2458 --- [ocp-async-6,d44f098511eca5b1,da4c44400615237d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:22.567 INFO 2458 --- [ocp-async-9,f155816a7d7e1c61,a6775dc4afb3d9cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:22.573 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:22:22.573 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:22:22.573 INFO 2458 --- [ocp-schedules-26,f204df0738449975,d9ec68bfa9e5b317] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:22:23.251 INFO 2458 --- [host-check-clockdiff5,d04df2f02874d049,7fc942d9b5c91605] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535343 1 1 , error= 2025-10-27 11:22:23.707 INFO 2458 --- [prometheus-minute-schedule-1,88effa6f7beeec36,253515311f545532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:23.717 INFO 2458 --- [prometheus-minute-schedule-1,88effa6f7beeec36,253515311f545532] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:24.308 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:24.309 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:24.567 WARN 2458 --- [ocp-async-4,d5130205f2e366ce,ebc1adeb46b28048] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:24.567 INFO 2458 --- [ocp-async-8,316e11cca5f744fa,755e900373fb3751] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:24.715 INFO 2458 --- [ocp-schedules-25,d7fee1ca00d4ce53,ab1b14957421acee] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:24.719 INFO 2458 --- [ocp-schedules-25,d7fee1ca00d4ce53,ab1b14957421acee] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:26.561 INFO 2458 --- [ocp-schedules-26,f0f37069aa9e175b,7c4d5e0c9e223709] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:26.567 WARN 2458 --- [ocp-async-2,12e23b1f4d76b279,e5ea13a0ea28b299] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:26.568 INFO 2458 --- [ocp-async-7,ee379003668a2cd0,6abf2e15bdce3323] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:26.707 INFO 2458 --- [prometheus-minute-schedule-1,e434871481f2d4d2,fa6fb9718001f45c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:26.714 INFO 2458 --- [prometheus-minute-schedule-1,e434871481f2d4d2,fa6fb9718001f45c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:27.229 INFO 2458 --- [ocp-schedules-2,3906d202cb50d938,de6e291158413812] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:22:27.237 INFO 2458 --- [ocp-schedules-2,3906d202cb50d938,de6e291158413812] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:27.237 INFO 2458 --- [ocp-schedules-2,3906d202cb50d938,de6e291158413812] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:27.309 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:27.310 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:27.740 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4b3e72c89a2e617e,089c30635accf35d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=4b3e72c89a2e617e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:22:27.740 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,4b3e72c89a2e617e,089c30635accf35d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=4b3e72c89a2e617e, duration=1 ms] 2025-10-27 11:22:28.523 INFO 2458 --- [ocp-async-7,cf1ddeac215cca54,c3f488b577193695] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:28.567 WARN 2458 --- [ocp-async-1,9d62218c8f1da4f7,e6501b106cafc0da] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:28.568 INFO 2458 --- [ocp-async-9,758e71f006f46e4d,177ca779262b17a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:28.967 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:22:28.967 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:22:28.969 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:22:28.969 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:22:28.970 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:22:28.970 INFO 2458 --- [ocp-schedules-14,dcd21f9cc8a3b11a,935b5959af106c37] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:22:29.416 INFO 2458 --- [ocp-schedules-11,c95212dae771ca06,edea1bd13d25e80d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:22:29.423 INFO 2458 --- [ocp-schedules-11,c95212dae771ca06,edea1bd13d25e80d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:22:29.423 INFO 2458 --- [ocp-schedules-11,c95212dae771ca06,edea1bd13d25e80d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:22:29.707 INFO 2458 --- [prometheus-minute-schedule-1,716207d747467b10,34415472d62ff287] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:29.719 INFO 2458 --- [prometheus-minute-schedule-1,716207d747467b10,34415472d62ff287] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:30.310 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:30.312 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:30.568 WARN 2458 --- [ocp-async-4,7e53594aa127f36e,bc950227393146d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:30.568 INFO 2458 --- [ocp-async-8,aab95d7ad0365c26,cca8efda87ffcc52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:30.612 INFO 2458 --- [ocp-schedules-7,42114269f38441a0,fd41ad1de87dbf35] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:30.647 WARN 2458 --- [scanner-112177679-1,42114269f38441a0,b8edd9759746687d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:30.673 INFO 2458 --- [alarm-detect-14,42114269f38441a0,5998214b10d8a6ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:30.681 INFO 2458 --- [alarm-detect-14,42114269f38441a0,5998214b10d8a6ec] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:30.687 INFO 2458 --- [alarm-detect-4,42114269f38441a0,521363a5a7ef5f9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:30.688 INFO 2458 --- [alarm-detect-18,42114269f38441a0,4efd00311f6bb09c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:30.693 INFO 2458 --- [alarm-detect-4,42114269f38441a0,521363a5a7ef5f9c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:30.698 INFO 2458 --- [alarm-detect-18,42114269f38441a0,4efd00311f6bb09c] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:30.752 INFO 2458 --- [alarm-detect-20,42114269f38441a0,7a235f7783d9e270] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:30.756 INFO 2458 --- [alarm-detect-20,42114269f38441a0,7a235f7783d9e270] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:30.787 INFO 2458 --- [alarm-detect-7,42114269f38441a0,6ff38ad0d4340c1f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535320 ~ 1761535340 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535335, value=115.3)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535335, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535335=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:22:31.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:31.713249+08:00, lastProcessTime=2025-10-27T11:22:21.719998+08:00 2025-10-27 11:22:31.934 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:31.713249+08:00 2025-10-27 11:22:32.568 WARN 2458 --- [ocp-async-2,b0d4611eb048754b,a6c2b209f36e4abc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:32.568 INFO 2458 --- [ocp-async-7,470ad742c96d765b,b6b8605435953376] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:32.707 INFO 2458 --- [prometheus-minute-schedule-1,9ffc3f2f8168e57f,9b9c1a5fcb375377] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:32.715 INFO 2458 --- [prometheus-minute-schedule-1,9ffc3f2f8168e57f,9b9c1a5fcb375377] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:33.312 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:33.312 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:33.871 INFO 2458 --- [ocp-schedules-28,9cc9143b54f90241,69677e0b4fd3eae6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:22:33.871 INFO 2458 --- [ocp-schedules-28,9cc9143b54f90241,69677e0b4fd3eae6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:22:34.568 WARN 2458 --- [ocp-async-1,190093e44da049a9,dd28e9dc18356c0e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:34.568 INFO 2458 --- [ocp-async-6,003200ba433f8d1d,97a8f746c694487c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:34.726 INFO 2458 --- [ocp-schedules-23,d6f1cb06f073c630,db18a03ecebdb885] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:34.730 INFO 2458 --- [ocp-schedules-23,d6f1cb06f073c630,db18a03ecebdb885] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:35.707 INFO 2458 --- [prometheus-minute-schedule-1,9bead15788890480,226c9a5107ad60dd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:35.715 INFO 2458 --- [prometheus-minute-schedule-1,9bead15788890480,226c9a5107ad60dd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:36.312 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:36.313 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:36.568 WARN 2458 --- [ocp-async-9,19ab7f43acc32a9b,b06ac0b5e7958a53] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:36.568 INFO 2458 --- [ocp-async-9,dcbcb626212251b6,3c354db25508d0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:36.574 INFO 2458 --- [ocp-schedules-21,23f9998541adba19,9a5b06bcc667fdee] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:37.708 INFO 2458 --- [prometheus-cache-monitor1,0421288f32944fc1,867bcb2945ef9d0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20393,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:22:37.708 INFO 2458 --- [prometheus-cache-monitor1,0421288f32944fc1,867bcb2945ef9d0e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:22:37.936 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:22:37.936 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:22:38.040 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:38.041 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:22:38.057 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:38.057 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:22:38.120 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:38.121 INFO 2458 --- [ocp-schedules-6,71e0f95a200ee8c8,b5f0fa90e719e135] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:22:38.523 INFO 2458 --- [ocp-async-9,38840cbd7adb2a50,88d5157433d8b0b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:38.568 INFO 2458 --- [ocp-async-2,d400e90e531ba6cb,d78410b40511a11e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:38.568 WARN 2458 --- [ocp-async-10,83b17851735509f0,a0376a9b4516de59] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:38.692 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:38.692 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:22:38.692 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:22:38.692 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:22:38.692 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:22:38.692 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:22:38.703 INFO 2458 --- [ocp-schedules-14,d32e20347b769663,4d59f9cbcb609bb4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:22:38.707 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:38.707 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:38.708 INFO 2458 --- [prometheus-minute-schedule-1,603ade59c7e8f883,67ee226e18ce771f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:38.708 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:38.708 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:38.709 INFO 2458 --- [ocp-schedules-14,d32e20347b769663,4d59f9cbcb609bb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:38.709 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:38.709 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:22:38.710 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:22:38.710 INFO 2458 --- [ocp-schedules-14,d32e20347b769663,4d59f9cbcb609bb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:22:38.710 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:38.712 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:38.712 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:38.712 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:38.713 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:38.715 INFO 2458 --- [prometheus-minute-schedule-1,603ade59c7e8f883,67ee226e18ce771f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:38.773 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:22:38.776 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:22:38.779 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:22:38.779 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:22:38.782 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:22:38.784 INFO 2458 --- [ocp-schedules-10,35b495e23bcd909a,1f15b9e75101e4ba] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:22:39.313 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:39.314 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:40.013 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:22:40.013 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:22:40.048 INFO 2458 --- [Check obproxy1,12969d092fe7e58f,33324152eb20fca7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:22:40.048 INFO 2458 --- [Check obproxy1,12969d092fe7e58f,33324152eb20fca7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:22:40.053 INFO 2458 --- [Collect obproxy1,7000c31aaecff9d5,c16de566aa151b71] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:22:40.053 INFO 2458 --- [Collect obproxy1,7000c31aaecff9d5,c16de566aa151b71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:22:40.078 INFO 2458 --- [Check obproxy1,12969d092fe7e58f,33324152eb20fca7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:22:40.081 INFO 2458 --- [Check obproxy1,12969d092fe7e58f,33324152eb20fca7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:22:40.081 INFO 2458 --- [Check obproxy1,12969d092fe7e58f,33324152eb20fca7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:22:40.082 INFO 2458 --- [Collect obproxy1,7000c31aaecff9d5,c16de566aa151b71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:22:40.084 INFO 2458 --- [Collect obproxy1,7000c31aaecff9d5,c16de566aa151b71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:22:40.087 INFO 2458 --- [Collect obproxy1,7000c31aaecff9d5,c16de566aa151b71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:22:40.568 WARN 2458 --- [ocp-async-3,6e28b1ff47cba149,1c7d7ad68f68df32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:40.568 INFO 2458 --- [ocp-async-1,a0ebfedd246b6c8d,0a65d3fe9d13382c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:40.794 INFO 2458 --- [ocp-schedules-18,6208c3d1fcce6767,2a3e914d0476a5b5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:40.834 WARN 2458 --- [scanner-112177926-1,6208c3d1fcce6767,d70dcd7dc98cbb2c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:40.903 INFO 2458 --- [alarm-detect-18,6208c3d1fcce6767,1e90e6147df577f1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535335 ~ 1761535350 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535350, value=117.018)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535350, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535350=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:22:41.085 INFO 2458 --- [ocp-schedules-13,296da06cfe385625,76ccb94b14039ce5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:22:41.085 INFO 2458 --- [ocp-schedules-13,296da06cfe385625,76ccb94b14039ce5] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:22:41.707 INFO 2458 --- [prometheus-minute-schedule-1,663e7c8fd5613136,6c6d4a240221d2da] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:41.715 INFO 2458 --- [prometheus-minute-schedule-1,663e7c8fd5613136,6c6d4a240221d2da] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:41.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:41.715889+08:00, lastProcessTime=2025-10-27T11:22:31.713249+08:00 2025-10-27 11:22:41.975 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:41.715889+08:00 2025-10-27 11:22:42.314 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:42.315 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:42.568 INFO 2458 --- [ocp-async-5,ad6829ac946c8d1a,7ef44c3591713bcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:42.568 WARN 2458 --- [ocp-async-6,af04b1c7d41fbda2,79603fede00a17d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:44.146 INFO 2458 --- [ocp-schedules-13,e1a199228f4833bf,cd8721ce652d41b2] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:22:44.568 WARN 2458 --- [ocp-async-7,d5739c36563b1816,863eac26d113d670] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:44.568 INFO 2458 --- [ocp-async-10,385f7e80d9e32928,dfd91d25805878fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:44.707 INFO 2458 --- [prometheus-minute-schedule-1,b1b23f04767af341,02f393a117aab146] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:44.716 INFO 2458 --- [prometheus-minute-schedule-1,b1b23f04767af341,02f393a117aab146] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:44.737 INFO 2458 --- [ocp-schedules-23,2b3067dad51f1982,9e650b3df4e57f7c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:44.741 INFO 2458 --- [ocp-schedules-23,2b3067dad51f1982,9e650b3df4e57f7c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:45.102 INFO 2458 --- [prometheus-second-schedule-1,c1aa8de0e53092ad,5839b87690dcd914] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:22:45.315 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:45.316 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:46.569 INFO 2458 --- [ocp-async-10,85afe45c1b74c6f7,60f61ec05b202bdd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:46.569 WARN 2458 --- [ocp-async-4,9d8d947be2b9e6e7,e5377bcc591bc804] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:46.581 INFO 2458 --- [ocp-schedules-20,3388d6677c0e13ca,d699412fee7e3e6a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:47.241 INFO 2458 --- [ocp-schedules-14,d577da015f20f0da,3d88d7e006d9536b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:22:47.249 INFO 2458 --- [ocp-schedules-14,d577da015f20f0da,3d88d7e006d9536b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:47.249 INFO 2458 --- [ocp-schedules-14,d577da015f20f0da,3d88d7e006d9536b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:22:47.333 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:22:47.360 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:47.364 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:47.416 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:47.418 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:47.467 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:47.469 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:47.518 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:47.521 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:47.573 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:47.575 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:47.624 INFO 2458 --- [ocp-schedules-12,e729cec2b8c46548,205cff008a6ee1d5] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:22:47.707 INFO 2458 --- [prometheus-minute-schedule-1,e610d44ddde3d143,d1da45b1fe683988] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:47.715 INFO 2458 --- [prometheus-minute-schedule-1,e610d44ddde3d143,d1da45b1fe683988] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:48.316 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:48.317 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:48.523 INFO 2458 --- [ocp-async-10,b32d710a5a82bbba,6894040e74854247] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:48.569 WARN 2458 --- [ocp-async-9,85eea9706374e36e,f376aa11b78c5fb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:48.569 INFO 2458 --- [ocp-async-6,96cef9f3f3fbae02,62e2707bed211c79] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:49.005 INFO 2458 --- [ocp-schedules-1,17c84c777835c94a,9db0d5de89182d9a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:22:49.005+08:00, clusters=[1, 2] 2025-10-27 11:22:49.026 INFO 2458 --- [ocp-schedules-1,17c84c777835c94a,9db0d5de89182d9a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:22:49.029 INFO 2458 --- [ocp-schedules-1,17c84c777835c94a,9db0d5de89182d9a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:22:49.944 INFO 2458 --- [ocp-schedules-15,d83c2be7c7c211c1,2788f1fccd794680] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:22:49.944 INFO 2458 --- [ocp-schedules-15,d83c2be7c7c211c1,2788f1fccd794680] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:22:49.951 INFO 2458 --- [ocp-schedules-15,d83c2be7c7c211c1,2788f1fccd794680] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:22:49.951 INFO 2458 --- [ocp-schedules-15,d83c2be7c7c211c1,2788f1fccd794680] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:22:50.569 INFO 2458 --- [ocp-async-5,6dfd88dbda57bdb6,d21e2fbebf0faca3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:50.569 WARN 2458 --- [ocp-async-8,37ebe12e14620c04,897593207ed56155] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:50.707 INFO 2458 --- [prometheus-minute-schedule-1,477172b3386681de,42d6aa51e6ab4f4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:50.716 INFO 2458 --- [prometheus-minute-schedule-1,477172b3386681de,42d6aa51e6ab4f4f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:50.909 INFO 2458 --- [ocp-schedules-17,8f7533f59a2e41e6,ccef82c6115c166d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:22:50.956 WARN 2458 --- [scanner-112178126-1,8f7533f59a2e41e6,fec313ea2829e5fc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:22:50.973 INFO 2458 --- [alarm-detect-20,8f7533f59a2e41e6,e8b13e9b8fc88d89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:50.978 INFO 2458 --- [alarm-detect-20,8f7533f59a2e41e6,e8b13e9b8fc88d89] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:22:50.988 INFO 2458 --- [alarm-detect-20,8f7533f59a2e41e6,fe612ce1f69a9d80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:50.992 INFO 2458 --- [alarm-detect-17,8f7533f59a2e41e6,6644f68ad852c410] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:50.995 INFO 2458 --- [alarm-detect-20,8f7533f59a2e41e6,fe612ce1f69a9d80] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:22:50.998 INFO 2458 --- [alarm-detect-17,8f7533f59a2e41e6,6644f68ad852c410] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:22:51.006 INFO 2458 --- [alarm-detect-2,8f7533f59a2e41e6,eb6d69d86747cec4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535300 ~ 1761535360 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535320, value=203654.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203654.56, values={1761535320=203654.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:22:51.046 INFO 2458 --- [alarm-detect-16,8f7533f59a2e41e6,9df4c1e23b542e54] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535301 ~ 1761535361 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535320, value=93.109)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.109, values={1761535320=93.109}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:22:51.058 INFO 2458 --- [alarm-detect-16,8f7533f59a2e41e6,e7868eddbc0da5b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:51.061 INFO 2458 --- [alarm-detect-6,8f7533f59a2e41e6,6d84b02469dea5a7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535301 ~ 1761535361 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535320, value=203655.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203655.912, values={1761535320=203655.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:22:51.063 INFO 2458 --- [alarm-detect-16,8f7533f59a2e41e6,e7868eddbc0da5b4] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:22:51.317 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:51.318 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:51.408 INFO 2458 --- [ocp-schedules-3,59fc530039dbfa57,8b3db06c3c35c557] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:51.408 INFO 2458 --- [ocp-schedules-10,8d9ffbac1d73c2cc,f614183bcd37e3bd] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:51.408 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:22:51.409 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:22:51.410 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:22:51.410 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:22:51.410 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:22:51.425 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:51.425 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:51.425 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:51.425 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:51.426 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:51.429 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:51.429 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:51.429 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:51.429 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:51.430 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:51.431 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:51.431 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:51.432 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:51.432 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:51.432 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:51.433 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:51.434 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:51.435 INFO 2458 --- [ocp-schedules-8,77e9b2df26bd39ff,c0edd05084dfbca9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:22:51.435 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:51.435 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:51.436 INFO 2458 --- [host-check-clockdiff1,77e9b2df26bd39ff,5d60ba5e64a4537f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:22:51.438 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:51.438 INFO 2458 --- [host-check-clockdiff2,77e9b2df26bd39ff,9b460411f9de97cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:22:51.441 INFO 2458 --- [host-check-clockdiff3,77e9b2df26bd39ff,2f355caa7af7552b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:22:51.442 INFO 2458 --- [host-check-clockdiff4,77e9b2df26bd39ff,8661797bb13de9d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:22:51.442 INFO 2458 --- [host-check-clockdiff5,77e9b2df26bd39ff,0dc728d6802246ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:22:51.442 INFO 2458 --- [ocp-schedules-14,bc4d62bd61df5b6e,490d768f0b5cec18] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:22:51.496 INFO 2458 --- [host-check-clockdiff4,77e9b2df26bd39ff,8661797bb13de9d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535371 -1 -1 , error= 2025-10-27 11:22:51.496 INFO 2458 --- [host-check-clockdiff1,77e9b2df26bd39ff,5d60ba5e64a4537f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535371 0 0 , error= 2025-10-27 11:22:51.503 INFO 2458 --- [host-check-clockdiff5,77e9b2df26bd39ff,0dc728d6802246ca] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535371 0 0 , error= 2025-10-27 11:22:51.520 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:22:51.522 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:22:51.523 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:22:51.521864569+08:00 2025-10-27 11:22:51.523 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:22:51.523+08:00 2025-10-27 11:22:51.524 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:22:51.524 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:22:51.524 INFO 2458 --- [host-check-agent-status5,8d9ffbac1d73c2cc,c7e547cb408c1270] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:22:51.529 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:22:51.529 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:22:51.531 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:22:51.530375445+08:00 2025-10-27 11:22:51.531 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:22:51.531+08:00 2025-10-27 11:22:51.533 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:22:51.533 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:22:51.533 INFO 2458 --- [host-check-agent-status3,8d9ffbac1d73c2cc,1359b12d4876653a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:22:51.543 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:22:51.523+08:00 and status AVAILABLE 2025-10-27 11:22:51.545 INFO 2458 --- [ocp-schedules-7,d075ff064dcf1214,4b0ef6b6393879ff] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:22:51.547 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:22:51.531+08:00 and status AVAILABLE 2025-10-27 11:22:51.548 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:22:51.548 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:22:51.549 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:22:51.547998789+08:00 2025-10-27 11:22:51.549 INFO 2458 --- [ocp-schedules-5,5b20193f9f6e844f,c955926a61bc51f3] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:22:51.550 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:22:51.549+08:00 2025-10-27 11:22:51.553 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:22:51.554 INFO 2458 --- [ocp-schedules-21,3d13032918d4c08a,893a3a0d1968146c] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:22:51.555 INFO 2458 --- [ocp-schedules-7,d075ff064dcf1214,4b0ef6b6393879ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:51.557 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:22:51.558 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:22:51.561 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:22:51.561 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:22:51.561 INFO 2458 --- [host-check-agent-status4,8d9ffbac1d73c2cc,4cb58b5196de2411] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:22:51.562 INFO 2458 --- [ocp-schedules-21,3d13032918d4c08a,893a3a0d1968146c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:51.562 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:22:51.562 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:22:51.562 INFO 2458 --- [host-check-agent-status1,8d9ffbac1d73c2cc,5a6451d6035a6bd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:22:51.565 INFO 2458 --- [ocp-schedules-5,5b20193f9f6e844f,c955926a61bc51f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:51.568 INFO 2458 --- [ocp-schedules-12,af719b1519def8b2,adc0c7fb55eee959] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:22:51.568 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:22:51.568 INFO 2458 --- [ocp-schedules-1,505be9ac8261babb,b63068a13c12cd23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:22:51.569 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:22:51.569 INFO 2458 --- [ocp-schedules-9,ec5d4ff1c388648b,f3a8d221a68d01e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:22:51.569 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:22:51.549+08:00 and status AVAILABLE 2025-10-27 11:22:51.570 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:22:51.569942838+08:00 2025-10-27 11:22:51.570 INFO 2458 --- [ocp-schedules-29,23e3ca59294b0d19,555a64637d76d3c2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:22:51.571 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:22:51.570+08:00 2025-10-27 11:22:51.576 INFO 2458 --- [ocp-schedules-29,23e3ca59294b0d19,555a64637d76d3c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:22:51.577 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:22:51.579 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:22:51.579 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:22:51.579 INFO 2458 --- [host-check-agent-status2,8d9ffbac1d73c2cc,306ef1885fd2ea00] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:22:51.582 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:22:51.570+08:00 and status AVAILABLE 2025-10-27 11:22:51.587 INFO 2458 --- [ocp-schedules-5,5b20193f9f6e844f,c955926a61bc51f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:22:51.588 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:22:51.589 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:22:51.588987997+08:00 2025-10-27 11:22:51.589 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:22:51.590 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:22:51.589+08:00 2025-10-27 11:22:51.590 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:22:51.593 INFO 2458 --- [ocp-schedules-9,ec5d4ff1c388648b,f3a8d221a68d01e1] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:22:51.594 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:22:51.597 INFO 2458 --- [ocp-schedules-1,505be9ac8261babb,b63068a13c12cd23] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:22:51.599 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:22:51.601 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:22:51.589+08:00 and status AVAILABLE 2025-10-27 11:22:51.603 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:22:51.606 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:22:51.610 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:22:51.613 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:22:51.617 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.618 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:51.619 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:22:51.620 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:51.624 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:51.627 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.627 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:51.636 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.636 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:22:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:22:51.636 INFO 2458 --- [ocp-schedules-7,d075ff064dcf1214,4b0ef6b6393879ff] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:22:51.639 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:51.644 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:51.644 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,72745ab730164ed1,36880771d59cfefd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=72745ab730164ed1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:22:51.644 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,72745ab730164ed1,36880771d59cfefd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=72745ab730164ed1, duration=1 ms] 2025-10-27 11:22:51.645 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.646 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:22:51.653 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.656 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:22:51.659 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.660 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:22:51.670 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:51.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.690 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:22:51.690 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7127b45925d140ad,36ceee94b99dc9d0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=7127b45925d140ad, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:22:51.690 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,7127b45925d140ad,36ceee94b99dc9d0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=7127b45925d140ad, duration=0 ms] 2025-10-27 11:22:51.692 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:51.693 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.693 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:51.694 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:22:51.695 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:51.712 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.713 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:22:51.715 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:22:51.715 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:22:51.716 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.719 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:22:51.720 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:22:51.712278+08:00, lastProcessTime=2025-10-27T11:22:41.715889+08:00 2025-10-27 11:22:51.723 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:22:51.724 INFO 2458 --- [ocp-schedules-29,23e3ca59294b0d19,555a64637d76d3c2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:22:51.729 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:51.733 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:22:51.734 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:22:51.738 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.739 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:51.751 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:22:51.761 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:22:51.762 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:51.762 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:22:51.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:22:51.782 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:22:51.792 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:51.795 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:22:51.799 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:51.799 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:22:51.807 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:22:51.813 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:51.819 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:22:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:51.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:51.825 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:22:51.829 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:51.831 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:51.834 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:22:51.848 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:22:51.861 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:22:51.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:51.864 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.867 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:22:51.869 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:22:51.872 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:22:51.878 INFO 2458 --- [ocp-schedules-27,29a884dc68b7e1a5,002b496eff002b80] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:22:51.878 INFO 2458 --- [ocp-schedules-27,29a884dc68b7e1a5,002b496eff002b80] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:22:51.882 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:51.882 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:51.890 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:22:51.893 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:22:51.909 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:22:51.909 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:22:51.911 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:51.923 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:51.924 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:51.929 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:51.938 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:22:51.943 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:51.943 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:22:51.944 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:22:51.946 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:51.947 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.948 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:51.949 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:22:51.949 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:22:51.960 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:22:51.963 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:22:51.973 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:22:51.984 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:22:51.986 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:22:51.990 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:51.991 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.991 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:22:51.993 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:51.993 INFO 2458 --- [host-check-host-status4,59fc530039dbfa57,62982a78957e9b30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:22:51.996 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:22:51.997 INFO 2458 --- [host-check-host-status2,59fc530039dbfa57,4cf4fdcaa9a34c0f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:22:51.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:52.001 INFO 2458 --- [host-check-host-status3,59fc530039dbfa57,a0e5d9d059623d17] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:22:52.009 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:22:51.712278+08:00 2025-10-27 11:22:52.016 INFO 2458 --- [host-check-host-status5,59fc530039dbfa57,9b69d2f3b3866523] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:22:52.016 INFO 2458 --- [host-check-host-status1,59fc530039dbfa57,14ed4ac90dae751d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:22:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:22:52.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.088 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:52.090 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:22:52.090 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.091 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:22:52.093 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.094 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:22:52.109 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:22:52.120 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:52.123 INFO 2458 --- [pool-overview-container-5,a01396cf3e8e5d2f,9621d68923ba7fed] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.130 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:52.131 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.131 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:22:52.153 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:22:52.153 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:22:52.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:22:52.192 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.192 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:22:52.232 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:22:52.238 INFO 2458 --- [pool-overview-container-6,b1ac73f3b1c3be15,2d8e066f3088536f] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.280 INFO 2458 --- [pool-overview-container-7,26270bf23e8f522d,998eb07d39b275d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.282 INFO 2458 --- [pool-overview-container-7,26270bf23e8f522d,998eb07d39b275d1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:22:52.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:52.297 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.297 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:22:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:22:52.305 INFO 2458 --- [pool-12-thread-15,b1ac73f3b1c3be15,055a55fa2995e23a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:22:52.305 INFO 2458 --- [pool-12-thread-15,b1ac73f3b1c3be15,055a55fa2995e23a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:22:52.320 INFO 2458 --- [pool-12-thread-13,b1ac73f3b1c3be15,dd0116ff36da5b21] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:22:52.321 INFO 2458 --- [pool-12-thread-16,b1ac73f3b1c3be15,17c83fda8e83ac40] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:22:52.321 INFO 2458 --- [pool-12-thread-14,b1ac73f3b1c3be15,5e370fd1747a50ff] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:22:52.321 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:22:52.334 INFO 2458 --- [ForkJoinPool-7-worker-2,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.334 INFO 2458 --- [ForkJoinPool-7-worker-4,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.336 INFO 2458 --- [ForkJoinPool-7-worker-11,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.337 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:22:52.337 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:22:52.338 INFO 2458 --- [ForkJoinPool-7-worker-13,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:22:52.339 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:22:52.339 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:22:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:22:52.344 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:22:52.352 INFO 2458 --- [pool-12-thread-13,b1ac73f3b1c3be15,dd0116ff36da5b21] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:22:52.352 INFO 2458 --- [pool-12-thread-13,b1ac73f3b1c3be15,dd0116ff36da5b21] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:22:52.370 INFO 2458 --- [pool-12-thread-15,b1ac73f3b1c3be15,055a55fa2995e23a] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:22:52.371 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.373 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.373 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:22:52.374 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:22:52.375 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.375 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:22:52.377 INFO 2458 --- [pool-12-thread-15,b1ac73f3b1c3be15,055a55fa2995e23a] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:22:52.385 INFO 2458 --- [host-check-clockdiff2,77e9b2df26bd39ff,9b460411f9de97cf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535372 -1 -1 , error= 2025-10-27 11:22:52.404 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:22:52.407 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.408 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:22:52.409 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.410 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:22:52.412 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.412 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:22:52.413 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:22:52.417 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.418 INFO 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:22:52.419 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.420 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:22:52.421 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.421 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.421 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:22:52.422 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:22:52.423 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:22:52.442 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.446 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:22:52.452 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.452 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:22:52.454 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:22:52.459 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.460 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:22:52, 2025-10-27 11:22:52] 2025-10-27 11:22:52.482 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:22:52.482 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.485 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:22:52.485 WARN 2458 --- [ForkJoinPool-7-worker-13,b1ac73f3b1c3be15,7faf9dfb66ea4c16] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:22:52.487 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.488 INFO 2458 --- [ForkJoinPool-7-worker-4,b1ac73f3b1c3be15,c55fcf4981d32924] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:22:52.494 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:22:52.494 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.495 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:22:52.497 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.498 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:22:52.500 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:22:52.500 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.500 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:22:52.501 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.502 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:22:52, 2025-10-27 11:22:52] 2025-10-27 11:22:52.502 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:22:52.504 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:22:52.504 INFO 2458 --- [ForkJoinPool-7-worker-2,b1ac73f3b1c3be15,4ad5992cb37984f0] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:22:52.504 INFO 2458 --- [pool-12-thread-16,b1ac73f3b1c3be15,17c83fda8e83ac40] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:22:52.505 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.506 INFO 2458 --- [ForkJoinPool-7-worker-11,b1ac73f3b1c3be15,c04234e8186e51d7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:22:52.537 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:22:52.540 INFO 2458 --- [pool-12-thread-14,b1ac73f3b1c3be15,5e370fd1747a50ff] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:22:52.545 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:52.549 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.551 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.554 INFO 2458 --- [refresh-tenant-resource-cache-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:22:52.554 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:22:52.569 INFO 2458 --- [ocp-async-5,d53afb960faa556c,b1647e5eb8762418] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:52.569 WARN 2458 --- [ocp-async-9,17c9925054162066,e85385790d88b158] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:52.581 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:52.581 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.582 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:22:52.637 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:22:52.637 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:22:52.637 INFO 2458 --- [ocp-schedules-21,3d13032918d4c08a,893a3a0d1968146c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:22:52.693 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:22:52.693 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:22:52.697 INFO 2458 --- [refresh-tenant-resource-cache-8,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:22:52.923 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.929 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandb 2025-10-27 11:22:52.943 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.949 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.956 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.965 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.967 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.968 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.969 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.971 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.972 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.979 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.984 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.985 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:22:52.987 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.992 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.996 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.997 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:52.998 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:22:52.999 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.000 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:22:53.169 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.170 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.183 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.185 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.186 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.187 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.188 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.190 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.191 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.192 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.193 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:16:52.385, end=2025-10-27T11:21:52.919} 2025-10-27 11:22:53.193 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:22:53.230 INFO 2458 --- [host-check-clockdiff3,77e9b2df26bd39ff,2f355caa7af7552b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535373 0 0 , error= 2025-10-27 11:22:53.707 INFO 2458 --- [prometheus-minute-schedule-1,53e4b369f881a369,0a369544e4dc4b07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:53.725 INFO 2458 --- [prometheus-minute-schedule-1,53e4b369f881a369,0a369544e4dc4b07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:22:54.318 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:54.319 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:54.569 WARN 2458 --- [ocp-async-1,54ce318fbd6758d8,2c964a47b0b9c90e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:54.569 INFO 2458 --- [ocp-async-8,8c29ae78cc15092f,fca20eca70e0e07d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:54.748 INFO 2458 --- [ocp-schedules-11,5f8fde2f45419e85,362706dc975b1f68] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:22:54.753 INFO 2458 --- [ocp-schedules-11,5f8fde2f45419e85,362706dc975b1f68] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:22:55.141 INFO 2458 --- [ocp-schedules-16,2b9a7e3987672c86,4d260928aa70e4de] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:22:56.569 WARN 2458 --- [ocp-async-4,7b85774d4baf073b,851ea4e0786af04e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:56.569 INFO 2458 --- [ocp-async-4,7b0be3980b0fb2e8,1f2ebaf696434441] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:56.588 INFO 2458 --- [ocp-schedules-26,dea93903c4ed421b,2aaffc4995f53208] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:22:56.707 INFO 2458 --- [prometheus-minute-schedule-1,0a2c5f31373ebeed,2759330816606051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:56.714 INFO 2458 --- [prometheus-minute-schedule-1,0a2c5f31373ebeed,2759330816606051] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:22:57.319 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:22:57.320 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:22:58.524 INFO 2458 --- [ocp-async-2,ffa0afc45268d3ce,4d33da4dce64d19d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:22:58.569 WARN 2458 --- [ocp-async-10,a4752c8c75c2b83a,60477b50fbb3d7f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:22:58.569 INFO 2458 --- [ocp-async-5,31416b59fee03673,02522ab2aa4fc2ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:22:59.389 INFO 2458 --- [ocp-schedules-3,82a34e5afe73fba2,f450d0397b1f32c6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:22:59.707 INFO 2458 --- [prometheus-minute-schedule-1,fb95171e4e811643,94241567eb4c7114] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535380, interval=60 2025-10-27 11:22:59.716 INFO 2458 --- [prometheus-minute-schedule-1,fb95171e4e811643,94241567eb4c7114] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:00.320 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:00.321 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:00.570 INFO 2458 --- [ocp-async-1,9682739349db298d,e2b233099da03f99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:00.570 WARN 2458 --- [ocp-async-8,8b584f985ac508c2,e345a83f3a04aba7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:01.086 INFO 2458 --- [ocp-schedules-3,43f86e494040cbc2,d47c04c2ac798502] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:01.252 WARN 2458 --- [scanner-112178389-1,43f86e494040cbc2,2af28683a54a12fb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:01.503 INFO 2458 --- [alarm-detect-14,43f86e494040cbc2,201563ffbdace29b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535350 ~ 1761535371 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535365, value=38.325)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535365, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535365=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:01.506 INFO 2458 --- [alarm-detect-14,43f86e494040cbc2,201563ffbdace29b] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:23:01.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:01.713379+08:00, lastProcessTime=2025-10-27T11:22:51.712278+08:00 2025-10-27 11:23:01.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:01.713379+08:00 2025-10-27 11:23:02.570 INFO 2458 --- [ocp-async-6,00069f5a7d8ac62d,cf34bb308f8fcc06] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:02.570 WARN 2458 --- [ocp-async-4,c1bf9381a0765cd0,7b6ebafbf9de539b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:02.707 INFO 2458 --- [prometheus-minute-schedule-1,1a104a1001de93c0,334f4255eef4de70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:02.716 INFO 2458 --- [prometheus-minute-schedule-1,1a104a1001de93c0,334f4255eef4de70] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:03.321 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:03.322 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:03.696 INFO 2458 --- [ocp-schedules-28,aefd0c96ee9b3ef5,73227b37ec37cbaa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:23:03.704 INFO 2458 --- [ocp-schedules-28,aefd0c96ee9b3ef5,73227b37ec37cbaa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:03.747 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:03.747 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:23:03.761 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:03.762 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:23:03.775 INFO 2458 --- [ocp-schedules-28,aefd0c96ee9b3ef5,73227b37ec37cbaa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:23:04.570 INFO 2458 --- [ocp-async-9,eb32d5c35466eba6,70f10e7cbca1faa4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:04.570 WARN 2458 --- [ocp-async-5,e451eec6576bb3a2,8864a0a4f316f14e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:04.817 INFO 2458 --- [ocp-schedules-30,f5076ed6d87bab7d,da3f1779dd498f15] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:04.821 INFO 2458 --- [ocp-schedules-30,f5076ed6d87bab7d,da3f1779dd498f15] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:05.707 INFO 2458 --- [prometheus-minute-schedule-1,5c9c0fdff45e3281,7051de3d2148689d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:05.716 INFO 2458 --- [prometheus-minute-schedule-1,5c9c0fdff45e3281,7051de3d2148689d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:06.322 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:06.324 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:06.570 INFO 2458 --- [ocp-async-8,207c1427a2ac4f97,d79e0532f0db66ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:06.570 WARN 2458 --- [ocp-async-7,956625600b92d521,25493200e3c67213] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:06.595 INFO 2458 --- [ocp-schedules-15,9ca89d0bc1406cb9,e942c41942cfb8f8] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:07.251 INFO 2458 --- [ocp-schedules-24,cd12e5b1246e99cf,0809bba5fc14566b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:23:07.261 INFO 2458 --- [ocp-schedules-24,cd12e5b1246e99cf,0809bba5fc14566b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:07.261 INFO 2458 --- [ocp-schedules-24,cd12e5b1246e99cf,0809bba5fc14566b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:07.549 INFO 2458 --- [ocp-schedules-8,5e7444884fb78701,490fdf34880ed0e5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:23:07.556 INFO 2458 --- [ocp-schedules-8,5e7444884fb78701,490fdf34880ed0e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:07.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:23:07.640 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.644 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:23:07.652 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.653 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.680 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.681 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:23:07.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.704 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:23:07.709 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.710 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.712 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.713 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.719 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.720 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.741 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.742 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.754 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.755 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.768 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.769 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.781 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.781 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:23:07.784 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.784 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.799 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.800 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:07.800 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.801 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:07.802 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.803 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:07.803 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.804 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:07.805 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.805 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:07.806 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.807 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:07.807 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:23:07.813 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:23:07.813 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.813 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.840 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.840 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:23:07.843 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.843 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:23:07.868 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:07.869 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.869 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:23:07.874 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:23:07.882 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:07.882 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:23:07.882 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:07.882 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:23:07.886 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:23:07.886 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:23:07.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.905 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:23:07.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.908 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:23:07.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:23:07.962 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:07.962 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:23:08.005 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.006 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:08.007 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:08.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.009 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:08.010 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.010 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:08.011 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.012 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:08.013 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.013 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:08.014 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.015 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:08.015 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.016 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:08.016 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.017 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:08.018 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.019 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:23:08.019 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.020 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:23:08.020 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.021 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:23:08.022 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:23:08.025 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:23:08.105 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:08.105 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:08.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:23:08.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:23:08.134 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:23:08.134 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:23:08.134 INFO 2458 --- [ocp-schedules-8,5e7444884fb78701,490fdf34880ed0e5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:23:08.524 INFO 2458 --- [ocp-async-8,5fbcc672c190b8e9,12b1b15a7c1573ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:08.570 INFO 2458 --- [ocp-async-3,acf62c48db89ac07,29cc1b7d3ce13614] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:08.570 WARN 2458 --- [ocp-async-10,af1293c935c06fde,999e2955ba5b9353] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:08.707 INFO 2458 --- [prometheus-minute-schedule-1,1eae9446a4acdb9d,7bd80fc74436defc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:08.714 INFO 2458 --- [prometheus-minute-schedule-1,1eae9446a4acdb9d,7bd80fc74436defc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:09.213 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,42d7611b4862c0b4,36b8ae8c0c5490c3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=42d7611b4862c0b4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:23:09.214 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,42d7611b4862c0b4,36b8ae8c0c5490c3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=42d7611b4862c0b4, duration=1 ms] 2025-10-27 11:23:09.324 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:09.324 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:10.013 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:23:10.013 INFO 2458 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:23:10.049 INFO 2458 --- [Check obproxy1,7c3de4e60c1a48d5,3619a6eb8d22d164] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:23:10.049 INFO 2458 --- [Check obproxy1,7c3de4e60c1a48d5,3619a6eb8d22d164] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:23:10.053 INFO 2458 --- [Collect obproxy1,6c478e5a490eb8fe,3418daa4c061af94] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:23:10.053 INFO 2458 --- [Collect obproxy1,6c478e5a490eb8fe,3418daa4c061af94] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:23:10.079 INFO 2458 --- [Check obproxy1,7c3de4e60c1a48d5,3619a6eb8d22d164] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:23:10.083 INFO 2458 --- [Collect obproxy1,6c478e5a490eb8fe,3418daa4c061af94] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:23:10.083 INFO 2458 --- [Check obproxy1,7c3de4e60c1a48d5,3619a6eb8d22d164] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:23:10.083 INFO 2458 --- [Check obproxy1,7c3de4e60c1a48d5,3619a6eb8d22d164] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:23:10.086 INFO 2458 --- [Collect obproxy1,6c478e5a490eb8fe,3418daa4c061af94] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:23:10.089 INFO 2458 --- [Collect obproxy1,6c478e5a490eb8fe,3418daa4c061af94] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:23:10.179 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:23:10.570 WARN 2458 --- [ocp-async-9,2c6a8f4b9c7b946b,78f757ed144a1fab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:10.570 INFO 2458 --- [ocp-async-2,3f174f000c498dcb,6442de13b20f95f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:10.792 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=1000, affectRows=1000 2025-10-27 11:23:10.796 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=994, validCount=992 2025-10-27 11:23:10.809 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134601996) Duplicate entry '835014129065861120-1761535260' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134601996) Duplicate entry '835014129065861120-1761535260' for key 'PRIMARY' 2025-10-27 11:23:10.844 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=1000, affectRows=1000 2025-10-27 11:23:10.848 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=999, validCount=995 2025-10-27 11:23:10.888 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=1000, affectRows=1000 2025-10-27 11:23:10.894 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=992, validCount=989 2025-10-27 11:23:10.948 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=1000, affectRows=1000 2025-10-27 11:23:10.952 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=1000, validCount=995 2025-10-27 11:23:10.971 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=97, affectRows=97 2025-10-27 11:23:10.972 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=95, validCount=95 2025-10-27 11:23:11.217 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535320, nodeCount=94, affectRows=94 2025-10-27 11:23:11.235 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535320, nodeCount=101, affectRows=101 2025-10-27 11:23:11.540 INFO 2458 --- [ocp-schedules-1,792fcd0f1ef51584,8987770660e19c50] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:11.592 WARN 2458 --- [scanner-112180095-1,792fcd0f1ef51584,2d0e92a2ec2910dd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:11.610 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,d03fa5ba07d6fa54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:23:11.615 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,d03fa5ba07d6fa54] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:23:11.625 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,05bd8c6c55370bd8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:23:11.626 INFO 2458 --- [alarm-detect-6,792fcd0f1ef51584,98422bb2f5341737] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:23:11.633 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,05bd8c6c55370bd8] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:23:11.633 INFO 2458 --- [alarm-detect-6,792fcd0f1ef51584,98422bb2f5341737] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:23:11.707 INFO 2458 --- [prometheus-minute-schedule-1,ca30527a7d24f4f3,2a964da7582f3415] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:11.714 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,147b513eaf5efeb0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:23:11.715 INFO 2458 --- [prometheus-minute-schedule-1,ca30527a7d24f4f3,2a964da7582f3415] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:11.715460+08:00, lastProcessTime=2025-10-27T11:23:01.713379+08:00 2025-10-27 11:23:11.725 INFO 2458 --- [alarm-detect-8,792fcd0f1ef51584,147b513eaf5efeb0] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:23:11.765 INFO 2458 --- [alarm-detect-6,792fcd0f1ef51584,be5e8efc857c51b3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535365 ~ 1761535381 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535380, value=40.628)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535380, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535380=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:11.960 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:11.715460+08:00 2025-10-27 11:23:12.324 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:12.325 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:12.570 WARN 2458 --- [ocp-async-1,8ac890f1dd47d96b,3c84b19d43beaa6b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:12.570 INFO 2458 --- [ocp-async-1,30c11f6b14b45467,6ceaaddee832c0b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:12.877 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535320, nodeCount=626, affectRows=626 2025-10-27 11:23:14.570 WARN 2458 --- [ocp-async-8,fdb7e8c329fb9919,ee86d26f8c71e142] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:14.570 INFO 2458 --- [ocp-async-6,bb95e21a319af961,84afccf39623a31b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:14.707 INFO 2458 --- [prometheus-minute-schedule-1,fb7f08f6490fe0ae,cc0f6b2770aa3326] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:14.715 INFO 2458 --- [prometheus-minute-schedule-1,fb7f08f6490fe0ae,cc0f6b2770aa3326] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:14.814 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535320, nodeCount=1000, affectRows=1000 2025-10-27 11:23:14.818 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535320, nodeCount=1000, validCount=987 2025-10-27 11:23:14.835 INFO 2458 --- [ocp-schedules-6,450ac60a4f8580e3,eb4565391ec63254] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:14.838 INFO 2458 --- [ocp-schedules-6,450ac60a4f8580e3,eb4565391ec63254] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:15.325 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:15.327 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:15.414 INFO 2458 --- [ocp-schedules-10,d3acfb2300d45b42,e2c36bda3826fe69] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:23:15.414+08:00, clusters=[1, 2] 2025-10-27 11:23:15.416 INFO 2458 --- [ocp-schedules-10,d3acfb2300d45b42,e2c36bda3826fe69] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:23:15.418 INFO 2458 --- [ocp-schedules-10,d3acfb2300d45b42,e2c36bda3826fe69] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:23:16.571 WARN 2458 --- [ocp-async-3,42e76aec9cd4bf36,cfa636e618424017] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:16.571 INFO 2458 --- [ocp-async-3,95dfbca2d3aeb6a1,871e6e85c4cf8742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:16.603 INFO 2458 --- [ocp-schedules-12,e15d63b6f984b267,7c24bf45a4de7b6e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:16.691 INFO 2458 --- [ocp-schedules-30,f107158afadbf488,220b1beadb1c8d5a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:23:16.691 INFO 2458 --- [ocp-schedules-30,f107158afadbf488,220b1beadb1c8d5a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:23:16.747 INFO 2458 --- [Collect all tenants major compactions1,f107158afadbf488,a6601ac37944658c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:23:16.748 INFO 2458 --- [Collect all tenants major compactions2,f107158afadbf488,2a9623d6729c5292] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:23:16.749 INFO 2458 --- [Collect all tenants major compactions3,f107158afadbf488,9dbd9dd32e9d37ff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:23:16.752 INFO 2458 --- [Collect all tenants major compactions4,f107158afadbf488,753124e2f9d4cfc9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:23:16.757 INFO 2458 --- [Collect all tenants major compactions5,f107158afadbf488,a07d6fcada496f87] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:23:16.760 INFO 2458 --- [Collect all tenants major compactions6,f107158afadbf488,33e346e7e6cb4292] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:23:16.788 INFO 2458 --- [Collect all tenants major compactions6,f107158afadbf488,33e346e7e6cb4292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.789 INFO 2458 --- [Collect all tenants major compactions6,f107158afadbf488,33e346e7e6cb4292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:23:16.791 INFO 2458 --- [Collect all tenants major compactions5,f107158afadbf488,a07d6fcada496f87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.793 INFO 2458 --- [Collect all tenants major compactions5,f107158afadbf488,a07d6fcada496f87] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:23:16.794 INFO 2458 --- [Collect all tenants major compactions6,f107158afadbf488,33e346e7e6cb4292] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:23:16.796 INFO 2458 --- [Collect all tenants major compactions5,f107158afadbf488,a07d6fcada496f87] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:23:16.805 INFO 2458 --- [Collect all tenants major compactions3,f107158afadbf488,9dbd9dd32e9d37ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.806 INFO 2458 --- [Collect all tenants major compactions1,f107158afadbf488,a6601ac37944658c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.806 INFO 2458 --- [Collect all tenants major compactions4,f107158afadbf488,753124e2f9d4cfc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.807 INFO 2458 --- [Collect all tenants major compactions3,f107158afadbf488,9dbd9dd32e9d37ff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:23:16.807 INFO 2458 --- [Collect all tenants major compactions1,f107158afadbf488,a6601ac37944658c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:23:16.808 INFO 2458 --- [Collect all tenants major compactions4,f107158afadbf488,753124e2f9d4cfc9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:23:16.809 INFO 2458 --- [Collect all tenants major compactions2,f107158afadbf488,2a9623d6729c5292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:16.810 INFO 2458 --- [Collect all tenants major compactions3,f107158afadbf488,9dbd9dd32e9d37ff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:23:16.811 INFO 2458 --- [Collect all tenants major compactions1,f107158afadbf488,a6601ac37944658c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:23:16.811 INFO 2458 --- [Collect all tenants major compactions4,f107158afadbf488,753124e2f9d4cfc9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:23:16.812 INFO 2458 --- [Collect all tenants major compactions2,f107158afadbf488,2a9623d6729c5292] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:23:16.815 INFO 2458 --- [Collect all tenants major compactions2,f107158afadbf488,2a9623d6729c5292] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:23:16.816 INFO 2458 --- [ocp-schedules-30,f107158afadbf488,220b1beadb1c8d5a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:23:17.707 INFO 2458 --- [prometheus-minute-schedule-1,328ad12a99af2dd4,d794a592f847211d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:17.714 INFO 2458 --- [prometheus-minute-schedule-1,328ad12a99af2dd4,d794a592f847211d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:18.327 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:18.328 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:18.524 INFO 2458 --- [ocp-async-3,8dde43d72ab5a40d,8482b52d22e1f0cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:18.571 WARN 2458 --- [ocp-async-4,dfa950feb36adc6a,331cceb7a1e10ea3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:18.571 INFO 2458 --- [ocp-async-7,fa2622bfa690ba0a,460c3d828a68d3ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:20.091 INFO 2458 --- [ocp-schedules-3,7fc536f5a1d51ccd,b04ec23dac50ada7] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:23:20.103 INFO 2458 --- [ocp-schedules-4,83f2d9d3bd0e23fe,8db0a4e9780de98e] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:23:20.103 INFO 2458 --- [ocp-schedules-4,83f2d9d3bd0e23fe,8db0a4e9780de98e] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:23:20.111 INFO 2458 --- [ocp-schedules-4,83f2d9d3bd0e23fe,8db0a4e9780de98e] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:23:20.111 INFO 2458 --- [ocp-schedules-4,83f2d9d3bd0e23fe,8db0a4e9780de98e] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:23:20.571 INFO 2458 --- [ocp-async-6,4b1625dfe312c53e,8f9b2a1b529f0827] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:20.571 WARN 2458 --- [ocp-async-10,a82e54c2d0201f8e,39cf43bda927a843] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:20.707 INFO 2458 --- [prometheus-minute-schedule-1,ca01eb2036c02c3c,154ac3f9495f714f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:20.718 INFO 2458 --- [prometheus-minute-schedule-1,ca01eb2036c02c3c,154ac3f9495f714f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:21.328 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:21.329 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:21.416 INFO 2458 --- [ocp-schedules-28,03ea007a3f3ef89b,4a7c79258b74eb82] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:21.416 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:23:21.416 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:23:21.417 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:23:21.417 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:23:21.417 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:23:21.424 INFO 2458 --- [ocp-schedules-30,261b24a2f6e9782f,ed3bd691bc6c766b] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:21.425 INFO 2458 --- [host-check-clockdiff1,261b24a2f6e9782f,cb0840e0a76a0b19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:23:21.425 INFO 2458 --- [host-check-clockdiff2,261b24a2f6e9782f,cddb46ad8c11eabd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:23:21.425 INFO 2458 --- [host-check-clockdiff3,261b24a2f6e9782f,1d3fc4468513b851] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:23:21.425 INFO 2458 --- [host-check-clockdiff4,261b24a2f6e9782f,bb5cf45400c5b13e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:23:21.426 INFO 2458 --- [host-check-clockdiff5,261b24a2f6e9782f,94309e9abd587e69] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:23:21.439 INFO 2458 --- [host-check-clockdiff1,261b24a2f6e9782f,cb0840e0a76a0b19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535401 0 0 , error= 2025-10-27 11:23:21.439 INFO 2458 --- [host-check-clockdiff5,261b24a2f6e9782f,94309e9abd587e69] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535401 0 0 , error= 2025-10-27 11:23:21.445 INFO 2458 --- [ocp-schedules-3,88d0f01b12114f96,d182719943be2ac3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:23:21.445 INFO 2458 --- [host-check-clockdiff2,261b24a2f6e9782f,cddb46ad8c11eabd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535401 -1 -1 , error= 2025-10-27 11:23:21.445 INFO 2458 --- [host-check-clockdiff3,261b24a2f6e9782f,1d3fc4468513b851] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535401 0 0 , error= 2025-10-27 11:23:21.452 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:21.457 INFO 2458 --- [host-check-clockdiff4,261b24a2f6e9782f,bb5cf45400c5b13e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535401 -1 -1 , error= 2025-10-27 11:23:21.457 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:21.457 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:21.457 INFO 2458 --- [ocp-schedules-25,309f91fc91cf90f4,3280e114c5dfc610] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:21.458 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:23:21.458 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:23:21.458 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:21.458 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:21.458 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:23:21.459 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:23:21.459 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:23:21.462 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:21.462 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:21.462 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:21.463 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:21.475 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:21.476 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:21.477 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:21.477 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:21.477 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:21.478 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:21.479 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:21.480 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:21.480 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:21.482 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:21.482 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:21.540 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:23:21.542 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:23:21.544 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:23:21.544 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:23:21.544 INFO 2458 --- [host-check-agent-status3,03ea007a3f3ef89b,3e10a93042f578cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:23:21.546 INFO 2458 --- [ocp-schedules-17,6725d91ed49b1dd5,819453b1befe341e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:23:21.548 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:23:21.548 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:23:21.548 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:23:21.549 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:23:21.547582583+08:00 2025-10-27 11:23:21.549 INFO 2458 --- [ocp-schedules-4,05840c5f204fe550,56bc2419777bef66] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:23:21.549 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:23:21.549+08:00 2025-10-27 11:23:21.549 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:23:21.549 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:23:21.549700478+08:00 2025-10-27 11:23:21.550 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:23:21.550 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:23:21.550 INFO 2458 --- [host-check-agent-status2,03ea007a3f3ef89b,1f80613cb8c7bdc8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:23:21.550 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:23:21.549261449+08:00 2025-10-27 11:23:21.550 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:23:21.550+08:00 2025-10-27 11:23:21.551 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:23:21.550+08:00 2025-10-27 11:23:21.552 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:23:21.552 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:23:21.552 INFO 2458 --- [host-check-agent-status4,03ea007a3f3ef89b,48ff619d11edb88f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:23:21.554 INFO 2458 --- [ocp-schedules-21,e6eb05900388a129,661fabc65ad8573d] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:23:21.556 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:23:21.559 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:23:21.549+08:00 and status AVAILABLE 2025-10-27 11:23:21.559 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:23:21.559 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:23:21.559 INFO 2458 --- [host-check-agent-status5,03ea007a3f3ef89b,10e100469239d606] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:23:21.562 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:23:21.550+08:00 and status AVAILABLE 2025-10-27 11:23:21.562 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:23:21.550+08:00 and status AVAILABLE 2025-10-27 11:23:21.562 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:23:21.563 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:23:21.562699628+08:00 2025-10-27 11:23:21.564 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:23:21.563+08:00 2025-10-27 11:23:21.565 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:23:21.566 INFO 2458 --- [ocp-schedules-16,d976d48f4d4d048b,af1714f8c0a5ae7a] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:23:21.567 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:23:21.568 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:23:21.568025866+08:00 2025-10-27 11:23:21.568 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:23:21.568+08:00 2025-10-27 11:23:21.568 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:23:21.569 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:23:21.574 INFO 2458 --- [ocp-schedules-17,6725d91ed49b1dd5,819453b1befe341e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:21.574 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:23:21.563+08:00 and status AVAILABLE 2025-10-27 11:23:21.576 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:23:21.568+08:00 and status AVAILABLE 2025-10-27 11:23:21.576 INFO 2458 --- [ocp-schedules-21,e6eb05900388a129,661fabc65ad8573d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:21.579 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:23:21.584 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:23:21.592 INFO 2458 --- [ocp-schedules-4,05840c5f204fe550,56bc2419777bef66] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:21.595 INFO 2458 --- [ocp-schedules-4,05840c5f204fe550,56bc2419777bef66] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:23:21.610 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:21.613 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:21.615 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:21.617 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:21.618 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:21.620 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:21.623 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:21.625 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:21.626 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:21.628 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:21.645 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:23:21.649 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:23:21.649 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:23:21.649 INFO 2458 --- [host-check-agent-status1,03ea007a3f3ef89b,84327a7893db02f4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:23:21.696 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:21.698 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:21.700 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.701 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:21.702 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:21.702 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:23:21.706 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:21.710 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:21.717 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:23:21.719 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:23:21.719 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:23:21.726 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.726 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:23:21.726 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:21.733 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:23:21.733 INFO 2458 --- [ocp-schedules-17,6725d91ed49b1dd5,819453b1befe341e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:23:21.737 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:23:21.770 INFO 2458 --- [ocp-schedules-23,9e3622d4676ba5dd,14fbf1ec81e0faa9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:21.807 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:23:21.810 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:23:21.810 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.811 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:23:21.811 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:23:21.811 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:21.812 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:23:21.846 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:21.844684+08:00, lastProcessTime=2025-10-27T11:23:11.715460+08:00 2025-10-27 11:23:21.856 WARN 2458 --- [scanner-112180372-1,9e3622d4676ba5dd,4d37f9ed29214526] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:21.909 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:21.911 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:21.912 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:21.913 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:21.916 INFO 2458 --- [alarm-detect-19,9e3622d4676ba5dd,0dd88d30ce37a2f6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761535365 ~ 1761535391 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761535380, value=1.77)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.77, values={1761535380=1.77}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:21.917 INFO 2458 --- [alarm-detect-19,9e3622d4676ba5dd,0dd88d30ce37a2f6] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:23:21.925 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:23:21.930 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:23:21.937 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:23:21.937 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:21.956 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:23:21.961 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:23:21.972 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:21.973 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.974 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:23:21.976 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:23:21.977 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:23:21.986 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:23:21.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:21.995 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:23:21.996 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:23:22.005 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:23:22.026 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:22.037 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:22.039 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:22.044 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:23:22.051 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:22.054 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:23:22.056 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:22.060 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:23:22.080 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:23:22.082 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:23:22.087 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.089 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.090 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:22.091 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.093 INFO 2458 --- [host-check-host-status4,309f91fc91cf90f4,cb73293008e0d395] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:23:22.094 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:23:22.098 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:22.105 INFO 2458 --- [host-check-host-status5,309f91fc91cf90f4,c5e2d56ee7cfd601] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:23:22.108 INFO 2458 --- [host-check-host-status3,309f91fc91cf90f4,39189047a18053d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:23:22.122 INFO 2458 --- [host-check-host-status1,309f91fc91cf90f4,03141ce64e054ace] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:23:22.122 INFO 2458 --- [host-check-host-status2,309f91fc91cf90f4,5aef1c09a4734c53] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:23:22.139 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.140 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.140 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:22.142 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.144 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.144 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:22.144 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:23:22.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:22.177 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.201 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.201 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:22.210 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.211 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.212 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:22.213 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:23:22.222 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:23:22.232 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:22.235 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:22.237 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:21.844684+08:00 2025-10-27 11:23:22.243 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.243 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:22.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:22.287 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:23:22.328 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.328 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:22.330 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:23:22.330 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.331 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:23:22.365 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.366 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.366 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:22.366 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:23:22.395 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:23:22.419 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.420 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.421 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:23:22.428 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:22.448 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:23:22.448 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:23:22.480 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.480 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.481 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:22.485 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:23:22.495 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:23:22.504 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:22.507 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:22.554 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:23:22.571 WARN 2458 --- [ocp-async-5,5626f6146a4af3c6,8fc0f7c523b843e8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:22.571 INFO 2458 --- [ocp-async-9,25e2aa91d2a198d0,c0a18f7ba2f6c4ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:22.624 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:23:22.625 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.626 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:23:22.628 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.629 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:23:22.677 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:23:22.722 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:22.722 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:22.723 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:23:22.765 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:23:22.765 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:23:22.765 INFO 2458 --- [ocp-schedules-21,e6eb05900388a129,661fabc65ad8573d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:23:23.707 INFO 2458 --- [prometheus-minute-schedule-1,aaab7dce9dcca22b,e8f2a5a3ad23d2a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:23.718 INFO 2458 --- [prometheus-minute-schedule-1,aaab7dce9dcca22b,e8f2a5a3ad23d2a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:24.329 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:24.330 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:24.571 WARN 2458 --- [ocp-async-3,edf6b294c879212d,f262cb4de1cadf36] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:24.571 INFO 2458 --- [ocp-async-3,790c776ffba86670,58f95dffe4e32707] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:24.859 INFO 2458 --- [ocp-schedules-24,dcd19c964c23d56d,2bd11333c566c336] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:24.863 INFO 2458 --- [ocp-schedules-24,dcd19c964c23d56d,2bd11333c566c336] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:26.098 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bda5561409358e75,a20416f081306002] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=bda5561409358e75, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:23:26.098 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,bda5561409358e75,a20416f081306002] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=bda5561409358e75, duration=0 ms] 2025-10-27 11:23:26.571 WARN 2458 --- [ocp-async-7,00c80d21b8a185bc,895494595bfff63a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:26.572 INFO 2458 --- [ocp-async-10,0cf3d889c3cd316a,a4fc63ba85c2b0b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:26.609 INFO 2458 --- [ocp-schedules-29,e70e5331a408a760,ac72c69ae8ed764c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:26.707 INFO 2458 --- [prometheus-minute-schedule-1,c61d50a91feb266c,dee47d75de06f29d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:26.715 INFO 2458 --- [prometheus-minute-schedule-1,c61d50a91feb266c,dee47d75de06f29d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:27.287 INFO 2458 --- [ocp-schedules-27,7136916528069e58,6786840572b25782] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:23:27.305 INFO 2458 --- [ocp-schedules-27,7136916528069e58,6786840572b25782] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:27.305 INFO 2458 --- [ocp-schedules-27,7136916528069e58,6786840572b25782] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:27.330 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:27.331 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:28.524 INFO 2458 --- [ocp-async-10,83118a461b5b65a1,61dbf047a9d73264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:28.572 WARN 2458 --- [ocp-async-5,761ab0ca7bb829eb,ca97c7531629f63e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:28.572 INFO 2458 --- [ocp-async-5,f4d0b59fdfdcbaf8,aa8d363319b185b0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:28.971 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:23:28.971 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:23:28.973 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:23:28.973 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:23:28.974 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:23:28.974 INFO 2458 --- [ocp-schedules-1,5e52693be6d072bf,2eae94776a65f7a9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:23:29.424 INFO 2458 --- [ocp-schedules-13,3da6a8a108e00130,28128aeca18b5119] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:23:29.442 INFO 2458 --- [ocp-schedules-13,3da6a8a108e00130,28128aeca18b5119] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:23:29.442 INFO 2458 --- [ocp-schedules-13,3da6a8a108e00130,28128aeca18b5119] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:23:29.707 INFO 2458 --- [prometheus-minute-schedule-1,f0d0f69344e9222b,f58a75e4850ba305] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:29.717 INFO 2458 --- [prometheus-minute-schedule-1,f0d0f69344e9222b,f58a75e4850ba305] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:30.331 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:30.332 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:30.572 WARN 2458 --- [ocp-async-4,32f4c38b190f29f8,45fef70dc9da90c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:30.572 INFO 2458 --- [ocp-async-3,782c0e5284f3fce4,e733778618f4d483] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:31.730 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:31.729571+08:00, lastProcessTime=2025-10-27T11:23:21.844684+08:00 2025-10-27 11:23:31.956 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:31.729571+08:00 2025-10-27 11:23:32.095 INFO 2458 --- [ocp-schedules-8,1b8c84657801d533,f788fc1c1dcef551] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:32.168 INFO 2458 --- [alarm-detect-6,1b8c84657801d533,3405889b300d8956] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:23:32.171 WARN 2458 --- [scanner-112180552-1,1b8c84657801d533,5828e7436d26b475] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:32.172 INFO 2458 --- [alarm-detect-6,1b8c84657801d533,3405889b300d8956] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_exists_expired_xa_trans,sql_alarm_type-trans_stat,trans_state-pending,trans_type-xa, startId: 230, data is empty. 2025-10-27 11:23:32.210 INFO 2458 --- [alarm-detect-12,1b8c84657801d533,b767fa191ee43c5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:23:32.212 INFO 2458 --- [alarm-detect-9,1b8c84657801d533,58cb92e3ab40fedc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:23:32.216 INFO 2458 --- [alarm-detect-12,1b8c84657801d533,b767fa191ee43c5a] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_large_trans_exist,sql_alarm_type-trans_stat, startId: 230, data is empty. 2025-10-27 11:23:32.217 INFO 2458 --- [alarm-detect-9,1b8c84657801d533,58cb92e3ab40fedc] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_expired_trans_exist,is_xa_trans-0,sql_alarm_type-trans_stat,trans_state-pending, startId: 230, data is empty. 2025-10-27 11:23:32.288 INFO 2458 --- [alarm-detect-18,1b8c84657801d533,22919b3459783702] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761535380 ~ 1761535402 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761535395, value=1.54)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.54, values={1761535395=1.54}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:32.328 INFO 2458 --- [alarm-detect-6,1b8c84657801d533,c3df5422850d0316] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:23:32.332 INFO 2458 --- [alarm-detect-6,1b8c84657801d533,c3df5422850d0316] c.o.o.a.c.d.metrics.SqlMetricsQuery : metric: ob_tenant_long_trans_exist,sql_alarm_type-trans_stat,trans_state-long, startId: 230, data is empty. 2025-10-27 11:23:32.354 INFO 2458 --- [alarm-detect-7,1b8c84657801d533,dfe29f2c49357506] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535380 ~ 1761535402 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535395, value=65.55)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535395, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535395=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:32.572 INFO 2458 --- [ocp-async-8,946362929857361b,4ae9416af7ae17c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:32.572 WARN 2458 --- [ocp-async-7,90fe571de7d4bf1f,e18e79084676d8fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:32.707 INFO 2458 --- [prometheus-minute-schedule-1,87d3d8c1f38639d3,ebc94c09da92a4fe] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:32.716 INFO 2458 --- [prometheus-minute-schedule-1,87d3d8c1f38639d3,ebc94c09da92a4fe] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:33.332 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:33.332 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:33.880 INFO 2458 --- [ocp-schedules-2,772401be0c97187c,6d2ccff995fb05e1] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:23:33.880 INFO 2458 --- [ocp-schedules-2,772401be0c97187c,6d2ccff995fb05e1] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:23:34.199 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:23:34.199 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:23:34.202 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:23:34.202 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:23:34.202 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:23:34.202 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:23:34.205 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:23:34.205 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:23:34.205 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:23:34.205 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:23:34.232 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:23:34.239 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:23:34.239 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:23:34.239 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:23:34.239 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:23:34.239 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:23:34.246 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:23:34.572 WARN 2458 --- [ocp-async-10,ed383e24bd44f73f,284a77deb4b76495] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:34.572 INFO 2458 --- [ocp-async-6,b5a788c3cd5e9c33,6e1fd7e88c1e9d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:34.626 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:23:34.870 INFO 2458 --- [ocp-schedules-12,59c5f7bc375f207e,2e7aae60ddfd4d75] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:34.874 INFO 2458 --- [ocp-schedules-12,59c5f7bc375f207e,2e7aae60ddfd4d75] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:34.990 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.784484872S 2025-10-27 11:23:34.990 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:23:35.079 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:23:35.079 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:23:35.130 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:23:35.178 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:23:35.178 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:23:35.178 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:23:35.178 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:23:35.178 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:23:35.179 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:23:35.186 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:23:35.301 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] o.s.boot.SpringApplication : Started application in 1.099 seconds (JVM running for 257883.728) 2025-10-27 11:23:35.302 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.099686479S 2025-10-27 11:23:35.644 INFO 2458 --- [ocp-schedules-20,370685c8155a556a,2f05a22af0ee17ff] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:23:35.707 INFO 2458 --- [prometheus-minute-schedule-1,c354718651396a25,cb945d7070a74005] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:35.716 INFO 2458 --- [prometheus-minute-schedule-1,c354718651396a25,cb945d7070a74005] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:36.333 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:36.334 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:36.572 INFO 2458 --- [ocp-async-2,5aa20bedcd3a403e,b3feecbfbdc68f23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:36.572 WARN 2458 --- [ocp-async-5,a7bbac5c06659ded,12465208160c0673] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:36.618 INFO 2458 --- [ocp-schedules-6,57288ab75bc2b779,a3ee1b77023bb25f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:37.708 INFO 2458 --- [prometheus-cache-monitor1,b4f282297eb806f1,df0c470941529ac9] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20391,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5096,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:23:37.708 INFO 2458 --- [prometheus-cache-monitor1,b4f282297eb806f1,df0c470941529ac9] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:23:38.126 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:23:38.126 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:23:38.218 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=root 2025-10-27 11:23:38.218 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:38.218 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:38.266 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:38.266 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:38.267 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:38.268 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:23:38.281 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:38.281 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:38.282 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:38.282 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:23:38.327 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:23:38.358 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:38.359 INFO 2458 --- [ocp-schedules-6,e40533be34672bc3,2697f1e79ae13875] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:23:38.524 INFO 2458 --- [ocp-async-1,0dbf2f43425ac921,abcabc6108c85392] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:38.572 INFO 2458 --- [ocp-async-7,93df424cc1c8bb2d,0b0d2d3d19f886c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:38.572 WARN 2458 --- [ocp-async-3,9ee21cf08f5ef5a9,3bea1ca8b172841f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:38.707 INFO 2458 --- [prometheus-minute-schedule-1,87c2c9ac619ad277,865f980776c1df41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:38.710 INFO 2458 --- [ocp-schedules-23,1ff239a8c78cd075,498cd778b6753edb] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:23:38.715 INFO 2458 --- [prometheus-minute-schedule-1,87c2c9ac619ad277,865f980776c1df41] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:38.717 INFO 2458 --- [ocp-schedules-23,1ff239a8c78cd075,498cd778b6753edb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:38.718 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:23:38.718 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:23:38.718 INFO 2458 --- [ocp-schedules-23,1ff239a8c78cd075,498cd778b6753edb] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:23:38.796 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:38.796 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:23:38.796 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:23:38.797 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:23:38.797 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:23:38.797 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:23:38.809 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:38.809 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:38.810 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:38.810 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:38.810 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:38.813 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:38.814 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:38.814 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:38.814 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:38.814 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:38.964 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:23:38.964 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:23:38.967 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:23:38.967 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:23:38.979 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:23:38.981 INFO 2458 --- [ocp-schedules-10,3ed3b994bbf29e27,4b26823f86b7ff93] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:23:39.338 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:39.339 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:40.011 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:23:40.043 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:23:40.064 INFO 2458 --- [Collect obproxy1,15c2b419eb39741c,c3ccd15c72069f95] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:23:40.068 INFO 2458 --- [Collect obproxy1,15c2b419eb39741c,c3ccd15c72069f95] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:23:40.073 INFO 2458 --- [Check obproxy1,cdbd7c66ba1184a6,3d75b8f7e0b30312] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:23:40.073 INFO 2458 --- [Check obproxy1,cdbd7c66ba1184a6,3d75b8f7e0b30312] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:23:40.098 INFO 2458 --- [Collect obproxy1,15c2b419eb39741c,c3ccd15c72069f95] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:23:40.101 INFO 2458 --- [Collect obproxy1,15c2b419eb39741c,c3ccd15c72069f95] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:23:40.103 INFO 2458 --- [Check obproxy1,cdbd7c66ba1184a6,3d75b8f7e0b30312] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:23:40.103 INFO 2458 --- [Collect obproxy1,15c2b419eb39741c,c3ccd15c72069f95] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:23:40.105 INFO 2458 --- [Check obproxy1,cdbd7c66ba1184a6,3d75b8f7e0b30312] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:23:40.105 INFO 2458 --- [Check obproxy1,cdbd7c66ba1184a6,3d75b8f7e0b30312] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:23:40.572 WARN 2458 --- [ocp-async-9,a5f6125d9f1e8a2b,d83e2b8cfdbb7e1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:40.572 INFO 2458 --- [ocp-async-10,a3bb9b97c1d35e67,60914a9937101962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:41.147 INFO 2458 --- [ocp-schedules-12,e05a889cb8863941,6bdb1ce3896388a5] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:23:41.147 INFO 2458 --- [ocp-schedules-12,e05a889cb8863941,6bdb1ce3896388a5] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:23:41.707 INFO 2458 --- [prometheus-minute-schedule-1,a2a3a5ac29da5922,0746ffcd20eceae5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:41.714 INFO 2458 --- [prometheus-minute-schedule-1,a2a3a5ac29da5922,0746ffcd20eceae5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:41.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:41.714777+08:00, lastProcessTime=2025-10-27T11:23:31.729571+08:00 2025-10-27 11:23:41.930 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:41.714777+08:00 2025-10-27 11:23:42.339 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:42.340 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:42.360 INFO 2458 --- [ocp-schedules-2,7f3f4e329f58b256,63547a531939a825] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:42.436 WARN 2458 --- [scanner-112180803-1,7f3f4e329f58b256,5944bfe01e5b4e28] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:42.511 INFO 2458 --- [alarm-detect-18,7f3f4e329f58b256,4b68e03d5cdc438b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535395 ~ 1761535412 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535410, value=112.707)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535410, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535410=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:23:42.573 INFO 2458 --- [ocp-async-5,cdc3784cf40b6785,8cd505b0d412d758] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:42.573 WARN 2458 --- [ocp-async-4,07e79f207acf6c19,550628e64c0faec6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:44.162 INFO 2458 --- [ocp-schedules-17,0ddd63834e86e6af,75329bd193c2e893] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:23:44.573 WARN 2458 --- [ocp-async-8,9cf5d8754f15dd5d,d3192719465e374a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:44.573 INFO 2458 --- [ocp-async-1,cbed94a6b519f706,85c45ed5e51c9073] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:44.707 INFO 2458 --- [prometheus-minute-schedule-1,a5a8b750039c8b98,08b2f3b8b8a3fbe5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:44.715 INFO 2458 --- [prometheus-minute-schedule-1,a5a8b750039c8b98,08b2f3b8b8a3fbe5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:44.882 INFO 2458 --- [ocp-schedules-21,cae35e09dc835d25,9fd1f6d71f6110da] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:44.885 INFO 2458 --- [ocp-schedules-21,cae35e09dc835d25,9fd1f6d71f6110da] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:45.102 INFO 2458 --- [prometheus-second-schedule-1,7a03d4f27fb615e3,169851a8883788df] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:23:45.340 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:45.341 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:46.573 WARN 2458 --- [ocp-async-7,e422d7fcac414079,6bf8a0e2cc855329] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:46.573 INFO 2458 --- [ocp-async-9,45f8699d88d0f7d3,577c805484530116] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:46.625 INFO 2458 --- [ocp-schedules-24,71eeeed8b03d896a,e449cda555fcc18a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:47.309 INFO 2458 --- [ocp-schedules-17,5efc0270b2b0263d,17b08fd169fe240b] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:23:47.317 INFO 2458 --- [ocp-schedules-17,5efc0270b2b0263d,17b08fd169fe240b] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:47.317 INFO 2458 --- [ocp-schedules-17,5efc0270b2b0263d,17b08fd169fe240b] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:23:47.707 INFO 2458 --- [prometheus-minute-schedule-1,b1f56eecc718af96,4f9d7096777ee2dc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:47.716 INFO 2458 --- [prometheus-minute-schedule-1,b1f56eecc718af96,4f9d7096777ee2dc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:48.341 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:48.342 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:48.524 INFO 2458 --- [ocp-async-9,1ef120e839c5f5dd,11ba9d879df769ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:48.573 WARN 2458 --- [ocp-async-2,cdf51a4b3f0ebf23,d1d958d7a57dfc75] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:48.573 INFO 2458 --- [ocp-async-5,f6f9c50a8365dfa0,d3f112d1fa131e7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:49.084 INFO 2458 --- [ocp-schedules-28,b2761b313b91d5ba,e28567c41d99abe6] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:23:49.084+08:00, clusters=[1, 2] 2025-10-27 11:23:49.102 INFO 2458 --- [ocp-schedules-28,b2761b313b91d5ba,e28567c41d99abe6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:23:49.136 INFO 2458 --- [ocp-schedules-28,b2761b313b91d5ba,e28567c41d99abe6] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster myoceanbase2, exclude result is empty. 2025-10-27 11:23:50.257 INFO 2458 --- [ocp-schedules-25,820056fd6afbd5a6,44f608ce333a0fea] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:23:50.257 INFO 2458 --- [ocp-schedules-25,820056fd6afbd5a6,44f608ce333a0fea] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:23:50.265 INFO 2458 --- [ocp-schedules-25,820056fd6afbd5a6,44f608ce333a0fea] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:23:50.265 INFO 2458 --- [ocp-schedules-25,820056fd6afbd5a6,44f608ce333a0fea] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:23:50.573 WARN 2458 --- [ocp-async-8,983ad0bf15e295b6,0746f737ee9db11b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:50.573 INFO 2458 --- [ocp-async-1,6a5ef14af2790ba7,6a1bdc56b2b21685] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:50.707 INFO 2458 --- [prometheus-minute-schedule-1,336c6dd3afb5a7bb,527e1af44f608f19] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:50.716 INFO 2458 --- [prometheus-minute-schedule-1,336c6dd3afb5a7bb,527e1af44f608f19] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:51.342 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:51.342 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:51.411 INFO 2458 --- [ocp-schedules-29,355696b416534b9b,8d1f5bc5b71e8282] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:51.411 INFO 2458 --- [ocp-schedules-24,ab7622641a51e5ff,1b68bd3ef48c51cd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:51.412 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:23:51.412 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:23:51.412 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:23:51.412 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:23:51.412 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:23:51.413 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:23:51.413 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:23:51.413 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:23:51.413 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:23:51.413 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:23:51.425 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:51.426 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:51.427 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:51.427 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:51.427 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:51.427 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:51.431 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:51.432 INFO 2458 --- [ocp-schedules-4,7698291b2e6614c1,e7304c3905cd69e4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:23:51.432 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:51.433 INFO 2458 --- [host-check-clockdiff1,7698291b2e6614c1,0d69f20b8973ae3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:23:51.433 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:51.433 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:51.433 INFO 2458 --- [host-check-clockdiff3,7698291b2e6614c1,bd1407e86a12a2d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:23:51.434 INFO 2458 --- [host-check-clockdiff2,7698291b2e6614c1,3b35b61a0f2254c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:23:51.434 INFO 2458 --- [host-check-clockdiff4,7698291b2e6614c1,f509a73ec51edd04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:23:51.434 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:51.434 INFO 2458 --- [host-check-clockdiff5,7698291b2e6614c1,7335eb32edde1f25] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:23:51.440 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:51.440 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:51.441 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:51.441 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:51.451 INFO 2458 --- [host-check-clockdiff1,7698291b2e6614c1,0d69f20b8973ae3f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535431 0 0 , error= 2025-10-27 11:23:51.450 INFO 2458 --- [ocp-schedules-22,b86f34d36412b1d0,9ef6f789ee47ba17] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:23:51.474 INFO 2458 --- [host-check-clockdiff3,7698291b2e6614c1,bd1407e86a12a2d1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535431 -1 -1 , error= 2025-10-27 11:23:51.489 INFO 2458 --- [host-check-clockdiff2,7698291b2e6614c1,3b35b61a0f2254c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535431 -2 -2 , error= 2025-10-27 11:23:51.494 INFO 2458 --- [host-check-clockdiff4,7698291b2e6614c1,f509a73ec51edd04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535431 0 0 , error= 2025-10-27 11:23:51.515 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:23:51.516 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:23:51.517 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:23:51.518 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:23:51.517487501+08:00 2025-10-27 11:23:51.519 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:23:51.518911370+08:00 2025-10-27 11:23:51.519 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:23:51.518+08:00 2025-10-27 11:23:51.519 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:23:51.519+08:00 2025-10-27 11:23:51.520 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:23:51.520 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:23:51.520 INFO 2458 --- [host-check-agent-status2,355696b416534b9b,adba369c6f1179fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:23:51.525 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:23:51.525 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:23:51.529 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:23:51.529 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:23:51.529 INFO 2458 --- [host-check-agent-status3,355696b416534b9b,12acc4d33e4945f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:23:51.532 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:23:51.532 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:23:51.518+08:00 and status AVAILABLE 2025-10-27 11:23:51.534 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:23:51.519+08:00 and status AVAILABLE 2025-10-27 11:23:51.535 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:23:51.535 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:23:51.535 INFO 2458 --- [host-check-agent-status1,355696b416534b9b,6d5294404a62111e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:23:51.540 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:23:51.540 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:23:51.540 INFO 2458 --- [host-check-agent-status5,355696b416534b9b,f0fb4460f0df0358] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:23:51.540 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:23:51.540 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:23:51.541 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:23:51.539740068+08:00 2025-10-27 11:23:51.541 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:23:51.541+08:00 2025-10-27 11:23:51.541 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:23:51.542 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:23:51.542 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:23:51.542526264+08:00 2025-10-27 11:23:51.543 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:23:51.542+08:00 2025-10-27 11:23:51.543 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:23:51.543 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:23:51.543 INFO 2458 --- [host-check-agent-status4,355696b416534b9b,d5c4675c861c8116] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:23:51.544 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:23:51.544 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:23:51.544 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:23:51.543997361+08:00 2025-10-27 11:23:51.545 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:23:51.544+08:00 2025-10-27 11:23:51.545 INFO 2458 --- [ocp-schedules-10,945aba22d2fa087a,e3b633b54e1019d5] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:23:51.549 INFO 2458 --- [ocp-schedules-11,90d809fae58922f3,58264257da0ee4ee] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:23:51.552 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:23:51.541+08:00 and status AVAILABLE 2025-10-27 11:23:51.553 INFO 2458 --- [ocp-schedules-10,945aba22d2fa087a,e3b633b54e1019d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:51.553 INFO 2458 --- [ocp-schedules-11,90d809fae58922f3,58264257da0ee4ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:51.553 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:23:51.544+08:00 and status AVAILABLE 2025-10-27 11:23:51.553 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:23:51.542+08:00 and status AVAILABLE 2025-10-27 11:23:51.554 INFO 2458 --- [ocp-schedules-11,90d809fae58922f3,58264257da0ee4ee] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:23:51.554 INFO 2458 --- [ocp-schedules-11,9554bf0f179e941f,7403f128679343cd] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:23:51.557 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:23:51.559 INFO 2458 --- [ocp-schedules-11,9554bf0f179e941f,7403f128679343cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:51.560 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:23:51.560 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:23:51.567 INFO 2458 --- [ocp-schedules-16,c662af14d4f86b70,9d03c96710f05eae] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:23:51.568 INFO 2458 --- [ocp-schedules-21,d2da345170e3feb2,bfa23e17b6542a28] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:23:51.574 INFO 2458 --- [ocp-schedules-2,b5cd84146d10744f,591a2acd9aaf8697] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:23:51.579 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:23:51.584 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:23:51.585 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:23:51.585 INFO 2458 --- [ocp-schedules-21,d2da345170e3feb2,bfa23e17b6542a28] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:23:51.586 INFO 2458 --- [ocp-schedules-2,b5cd84146d10744f,591a2acd9aaf8697] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:23:51.586 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:51.588 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:23:51.589 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:51.590 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:51.593 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:23:51.593 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:23:51.601 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:23:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.635 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:51.646 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:23:51.646 INFO 2458 --- [ocp-schedules-10,945aba22d2fa087a,e3b633b54e1019d5] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:23:51.657 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.657 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:51.663 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:51.666 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:51.673 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:51.675 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:51.677 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:23:51.680 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:23:51.685 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:23:51.696 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:23:51.697 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:23:51.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.703 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:23:51.704 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.704 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.705 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.705 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:51.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:23:51.717633+08:00, lastProcessTime=2025-10-27T11:23:41.714777+08:00 2025-10-27 11:23:51.728 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:51.733 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.734 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:23:51.736 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:23:51.739 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:23:51.740 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:23:51.745 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:23:51.750 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:23:51.792 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:51.793 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:51.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:51.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.794 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:51.794 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.794 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.794 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.794 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:51.795 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:23:51.799 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:23:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:51.806 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.807 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:51.807 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:23:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.814 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:23:51.814 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:23:51.815 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:23:51.816 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:51.817 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:23:51.823 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:23:51.843 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:51.843 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:51.847 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:23:51.850 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:23:51.852 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:23:51.853 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:23:51.855 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:23:51.880 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:23:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.887 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:51.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:23:51.888 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:51.889 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:51.890 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:51.891 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:51.895 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:23:51.905 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:23:51.907 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:23:51.909 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:23:51.912 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:23:51.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:51.920 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:51.921 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:51.922 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:23:51.927 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:23:51.933 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:23:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:51.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:23:51.944 INFO 2458 --- [host-check-host-status3,ab7622641a51e5ff,8573b248009a6737] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:23:51.946 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:23:51.946 INFO 2458 --- [host-check-host-status2,ab7622641a51e5ff,a69232c61d035d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:23:51.949 INFO 2458 --- [host-check-host-status5,ab7622641a51e5ff,bf595a141710f255] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:23:51.955 INFO 2458 --- [host-check-host-status1,ab7622641a51e5ff,1b79de0bec13c6fe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:23:51.957 INFO 2458 --- [host-check-host-status4,ab7622641a51e5ff,933b45d1d3593050] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:23:51.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:51.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:52.007 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:23:52.021 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:23:51.717633+08:00 2025-10-27 11:23:52.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:52.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:52.049 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:23:52.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:23:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:23:52.074 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:23:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:52.101 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:23:52.102 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:23:52.125 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:23:52.125 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:23:52.147 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:52.147 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:23:52.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.224 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:23:52.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:23:52.237 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:23:52.246 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:23:52.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:23:52.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:23:52.349 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:23:52.349 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.350 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.352 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.353 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:23:52.355 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.355 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.355 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.356 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:23:52.387 INFO 2458 --- [host-check-clockdiff5,7698291b2e6614c1,7335eb32edde1f25] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535432 0 0 , error= 2025-10-27 11:23:52.396 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:23:52.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:23:52.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.438 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.439 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:23:52.439 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:23:52.439 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:23:52.440 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:23:52.483 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:23:52.483 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:23:52.484 INFO 2458 --- [ocp-schedules-11,9554bf0f179e941f,7403f128679343cd] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:23:52.517 INFO 2458 --- [ocp-schedules-13,e94b30f447db5518,510d290bf809ebac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:23:52.573 WARN 2458 --- [ocp-async-10,b1d9d5da6c0318e7,2246330ac457e02f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:52.573 INFO 2458 --- [ocp-async-10,3fefd1ba38d37001,4bcd5717edb9ffa5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:52.588 WARN 2458 --- [scanner-112180972-1,e94b30f447db5518,3df788be72abdd65] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:23:52.650 INFO 2458 --- [alarm-detect-7,e94b30f447db5518,117457b18f69d605] c.o.o.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, activeAt=2025-10-27T11:23:21.907+08:00 2025-10-27 11:23:52.677 INFO 2458 --- [alarm-detect-6,e94b30f447db5518,72e0b2e11b7f23b2] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535362 ~ 1761535422 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535380, value=203714.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203714.56, values={1761535380=203714.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:23:52.677 INFO 2458 --- [alarm-detect-6,e94b30f447db5518,72e0b2e11b7f23b2] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:23:52.734 INFO 2458 --- [alarm-detect-7,e94b30f447db5518,2bcd2e115eba3d78] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535362 ~ 1761535422 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535380, value=203715.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203715.912, values={1761535380=203715.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:23:52.735 INFO 2458 --- [alarm-detect-7,e94b30f447db5518,2bcd2e115eba3d78] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:23:52.753 INFO 2458 --- [alarm-detect-17,e94b30f447db5518,aec406c68c94d6d6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535362 ~ 1761535422 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535380, value=93.118)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.118, values={1761535380=93.118}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:23:52.754 INFO 2458 --- [alarm-detect-17,e94b30f447db5518,aec406c68c94d6d6] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:23:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:23:53.234 INFO 2458 --- [ocp-schedules-7,8c636f87227d5955,463b631302fbdbdc] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20391,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7885} 2025-10-27 11:23:53.234 INFO 2458 --- [ocp-schedules-7,8c636f87227d5955,463b631302fbdbdc] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:23:53.707 INFO 2458 --- [prometheus-minute-schedule-1,ed1b1b965f803fe4,a4f62c818285df05] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:53.715 INFO 2458 --- [prometheus-minute-schedule-1,ed1b1b965f803fe4,a4f62c818285df05] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:54.343 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:54.343 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:54.573 WARN 2458 --- [ocp-async-4,1d2a9634549abfc8,3d6128ed8ca6207e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:54.573 INFO 2458 --- [ocp-async-8,ca6b870cd0f9de93,eed68a2731c91c51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:54.895 INFO 2458 --- [ocp-schedules-15,ac1069d684069044,d900b8f52345d1a1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:23:54.899 INFO 2458 --- [ocp-schedules-15,ac1069d684069044,d900b8f52345d1a1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:23:55.187 INFO 2458 --- [ocp-schedules-30,78f12ede695b04c2,a9fd1378cae9fd52] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:23:56.574 WARN 2458 --- [ocp-async-3,724612b8f250c60e,e55a099250e9cabe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:56.574 INFO 2458 --- [ocp-async-7,6ec37e72fea57780,9321ebe31cafdb75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:56.632 INFO 2458 --- [ocp-schedules-29,b01ba5ee71805cc0,76d966178017b90e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:23:56.707 INFO 2458 --- [prometheus-minute-schedule-1,e8a7c3c7611da806,2b0e6b917a4074f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:56.714 INFO 2458 --- [prometheus-minute-schedule-1,e8a7c3c7611da806,2b0e6b917a4074f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:23:57.344 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:23:57.344 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:23:58.140 INFO 2458 --- [ocp-schedules-26,e2ddad0826b348bd,323fdb7e56e68241] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:23:58.148 INFO 2458 --- [ocp-schedules-26,e2ddad0826b348bd,323fdb7e56e68241] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:23:58.190 INFO 2458 --- [ocp-schedules-26,e2ddad0826b348bd,323fdb7e56e68241] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:23:58.525 INFO 2458 --- [ocp-async-7,edf7ed7684bd20e2,a281b35b90bdb4bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:23:58.574 WARN 2458 --- [ocp-async-9,5424bef5ece45299,d3bf594967b1a823] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:23:58.574 INFO 2458 --- [ocp-async-9,dcb0311df5d5923d,0f74ebf7f6fc6fb0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:23:59.403 INFO 2458 --- [ocp-schedules-8,2a09bdc6aaf0a9dc,969cc207fa69683c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:23:59.707 INFO 2458 --- [prometheus-minute-schedule-1,5350be1d3ce3060f,6ef0b690b2aaf1a4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535440, interval=60 2025-10-27 11:23:59.716 INFO 2458 --- [prometheus-minute-schedule-1,5350be1d3ce3060f,6ef0b690b2aaf1a4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:00.344 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:00.345 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:00.574 WARN 2458 --- [ocp-async-4,2b150a5adf24feb8,463076485e4e3bb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:00.574 INFO 2458 --- [ocp-async-4,930c7b68bafda9c6,e26e21ac4e36292f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:01.726 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:01.724789+08:00, lastProcessTime=2025-10-27T11:23:51.717633+08:00 2025-10-27 11:24:02.291 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:01.724789+08:00 2025-10-27 11:24:02.574 WARN 2458 --- [ocp-async-4,bf57d8e74ae861b8,76e1fc05e4c04c52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:02.574 INFO 2458 --- [ocp-async-3,aae6bed711420853,4fa8c571b14b68bc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:02.707 INFO 2458 --- [prometheus-minute-schedule-1,ee5bbb5b39a143bf,f05a438bf48a9d4e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:02.727 INFO 2458 --- [prometheus-minute-schedule-1,ee5bbb5b39a143bf,f05a438bf48a9d4e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:02.784 INFO 2458 --- [ocp-schedules-28,e471dbcd102a9482,c8ea41676fc4a183] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:02.853 WARN 2458 --- [scanner-112181227-1,e471dbcd102a9482,148d2653d9c8dffc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:03.303 INFO 2458 --- [alarm-detect-5,e471dbcd102a9482,1d26c5726684ce8a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535410 ~ 1761535433 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535425, value=100.98)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535425, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535425=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:24:03.346 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:03.346 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:03.775 INFO 2458 --- [ocp-schedules-27,f332ac26f7f74cbd,a8787ede98a8e375] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:24:03.783 INFO 2458 --- [ocp-schedules-27,f332ac26f7f74cbd,a8787ede98a8e375] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:03.861 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:03.862 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:24:03.886 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:03.887 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:24:03.894 INFO 2458 --- [ocp-schedules-27,f332ac26f7f74cbd,a8787ede98a8e375] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:24:04.574 INFO 2458 --- [ocp-async-6,210d6304ba923fbb,3f531bd9a5ac6e87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:04.574 WARN 2458 --- [ocp-async-10,b3da7e5eb9a3b708,12fdad1df5c9aaa5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:04.906 INFO 2458 --- [ocp-schedules-4,d34dd1d2aa336cb3,558a70fb08c6ec11] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:04.910 INFO 2458 --- [ocp-schedules-4,d34dd1d2aa336cb3,558a70fb08c6ec11] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:05.707 INFO 2458 --- [prometheus-minute-schedule-1,8fb6161b7a9c73e2,56f23056e05bd52a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:05.718 INFO 2458 --- [prometheus-minute-schedule-1,8fb6161b7a9c73e2,56f23056e05bd52a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:06.023 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:06.024 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:24:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:24:06.028 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:24:06.108 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:24:06.150 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:24:06.174 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:24:06.174 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:24:06.174 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:24:06.174 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:24:06.177 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:24:06.177 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:24:06.177 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:24:06.177 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:24:06.178 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:06.178 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:24:06.180 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:06.180 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:24:06.180 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:24:06.180 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:24:06.254 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:21:30, start=2025-10-27T11:17:30, end=2025-10-27T11:20:00, updates=219, granule=120000ms, delay=240000ms 2025-10-27 11:24:06.261 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:21:30, target=2025-10-27T11:20:00, granule=120000ms, delay=90000ms) 2025-10-27 11:24:06.261 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:20:00, target=2025-10-27T11:12:30, granule=600000ms, delay=450000ms) 2025-10-27 11:24:06.261 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=233ms, period=120000ms 2025-10-27 11:24:06.346 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:06.348 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:06.574 INFO 2458 --- [ocp-async-6,e7f99c8d287467bc,a5aef1aafd2693b2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:06.574 WARN 2458 --- [ocp-async-5,9cfadb0ec2ff4432,c1cd2b801e300210] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:06.641 INFO 2458 --- [ocp-schedules-26,046549f540ef52ba,e9f9a35190c14f20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:07.319 INFO 2458 --- [ocp-schedules-11,f715e3c7853b1bbc,d864ce405e9f1708] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:24:07.327 INFO 2458 --- [ocp-schedules-11,f715e3c7853b1bbc,d864ce405e9f1708] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:07.327 INFO 2458 --- [ocp-schedules-11,f715e3c7853b1bbc,d864ce405e9f1708] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:07.577 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,3d33331aa51d3a4d,c0451fa33edd356f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=3d33331aa51d3a4d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:24:07.577 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,3d33331aa51d3a4d,c0451fa33edd356f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=3d33331aa51d3a4d, duration=1 ms] 2025-10-27 11:24:08.134 INFO 2458 --- [ocp-schedules-13,32553201f12622a5,52c34954485aef45] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:24:08.141 INFO 2458 --- [ocp-schedules-13,32553201f12622a5,52c34954485aef45] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:08.241 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:24:08.242 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.246 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:24:08.258 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.259 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.285 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.285 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.286 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:24:08.286 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.290 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:24:08.309 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.310 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.318 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.318 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.320 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.321 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.342 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.351 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.351 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.372 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.372 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:24:08.375 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.376 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.378 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.379 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.413 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.414 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.417 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.418 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.418 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.419 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.420 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.421 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.422 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.427 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.427 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.428 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.429 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.430 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.430 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:24:08.443 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.443 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:24:08.445 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.446 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:24:08.450 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:24:08.479 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.480 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:24:08.510 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.510 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:24:08.516 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.516 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:24:08.519 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:08.520 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.520 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:24:08.525 INFO 2458 --- [ocp-async-6,d171322128f6b7de,d653c3fedb0a0089] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:08.533 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.533 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:24:08.533 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.533 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:24:08.538 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:24:08.538 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:24:08.543 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.543 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:24:08.573 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.574 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:24:08.574 WARN 2458 --- [ocp-async-3,242de21e67ce0562,af679f3ec6357198] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:08.574 INFO 2458 --- [ocp-async-7,0d929a9c83647fce,8815760ffea8f566] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:08.628 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.629 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.630 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.631 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.632 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.634 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.634 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.635 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.636 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.637 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.637 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.638 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.639 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.640 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.640 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.641 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.642 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.643 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:24:08.644 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.644 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:24:08.645 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.646 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:24:08.647 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:24:08.650 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:24:08.707 INFO 2458 --- [prometheus-minute-schedule-1,5d14c3811c39bcad,613b300953e5fd37] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:08.721 INFO 2458 --- [prometheus-minute-schedule-1,5d14c3811c39bcad,613b300953e5fd37] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:08.748 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:08.749 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:08.749 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:24:08.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:24:08.788 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:24:08.788 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:24:08.788 INFO 2458 --- [ocp-schedules-13,32553201f12622a5,52c34954485aef45] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:24:09.349 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:09.350 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:10.017 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:24:10.017 INFO 2458 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:24:10.091 INFO 2458 --- [Check obproxy1,90655a28dc15d137,c250a57ee8ce889a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:24:10.091 INFO 2458 --- [Check obproxy1,90655a28dc15d137,c250a57ee8ce889a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:24:10.091 INFO 2458 --- [Collect obproxy1,56f07c583aaa2b9e,1f3c3b54927b49e9] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:24:10.091 INFO 2458 --- [Collect obproxy1,56f07c583aaa2b9e,1f3c3b54927b49e9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:24:10.121 INFO 2458 --- [Collect obproxy1,56f07c583aaa2b9e,1f3c3b54927b49e9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:24:10.121 INFO 2458 --- [Check obproxy1,90655a28dc15d137,c250a57ee8ce889a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:24:10.124 INFO 2458 --- [Collect obproxy1,56f07c583aaa2b9e,1f3c3b54927b49e9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:24:10.125 INFO 2458 --- [Check obproxy1,90655a28dc15d137,c250a57ee8ce889a] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:24:10.125 INFO 2458 --- [Check obproxy1,90655a28dc15d137,c250a57ee8ce889a] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:24:10.127 INFO 2458 --- [Collect obproxy1,56f07c583aaa2b9e,1f3c3b54927b49e9] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:24:10.182 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:24:10.575 INFO 2458 --- [ocp-async-7,9029611112f7b6ff,757a656b480c78ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:10.575 WARN 2458 --- [ocp-async-2,aa9d1f4b011cb178,d1d9945d9274292f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:11.046 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=1000, affectRows=1000 2025-10-27 11:24:11.051 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=994, validCount=992 2025-10-27 11:24:11.143 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=1000, affectRows=1000 2025-10-27 11:24:11.147 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=999, validCount=995 2025-10-27 11:24:11.162 ERROR 2458 --- [metric-archive-second-6,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134521644) Duplicate entry '835370602979598336-1761535320' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134521644) Duplicate entry '835370602979598336-1761535320' for key 'PRIMARY' 2025-10-27 11:24:11.305 INFO 2458 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535380, nodeCount=817, affectRows=817 2025-10-27 11:24:11.707 INFO 2458 --- [prometheus-minute-schedule-1,9eab12139f921252,11c620e1c18dd7ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:11.714 INFO 2458 --- [prometheus-minute-schedule-1,9eab12139f921252,11c620e1c18dd7ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:11.714856+08:00, lastProcessTime=2025-10-27T11:24:01.724789+08:00 2025-10-27 11:24:11.924 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=1000, affectRows=1000 2025-10-27 11:24:11.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:11.714856+08:00 2025-10-27 11:24:11.929 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=996, validCount=992 2025-10-27 11:24:12.007 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=1000, affectRows=1000 2025-10-27 11:24:12.012 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=996, validCount=992 2025-10-27 11:24:12.028 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=97, affectRows=97 2025-10-27 11:24:12.029 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=97, validCount=97 2025-10-27 11:24:12.350 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:12.351 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:12.575 INFO 2458 --- [ocp-async-7,2a16a2a71af94b11,a2f491e12bea2b3c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:12.575 WARN 2458 --- [ocp-async-8,2ea60f5efc298aa1,38e3c42483ed2af8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:13.312 INFO 2458 --- [ocp-schedules-10,21569b40e169c2f8,3836db51a7aa6cb4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:13.360 WARN 2458 --- [scanner-112182951-1,21569b40e169c2f8,8d7bac49f070cef5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:13.516 INFO 2458 --- [alarm-detect-4,21569b40e169c2f8,07458bd36e988c7e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535425 ~ 1761535443 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535440, value=54.278)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535440, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535440=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:24:14.575 WARN 2458 --- [ocp-async-3,5c9e5941e223a200,e6a67cde7ca9584c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:14.575 INFO 2458 --- [ocp-async-3,cebe1cfea4358a97,3e0422bb59a0e67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:14.707 INFO 2458 --- [prometheus-minute-schedule-1,956c575bebb4a6a9,0697f1be3ae55fd1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:14.716 INFO 2458 --- [prometheus-minute-schedule-1,956c575bebb4a6a9,0697f1be3ae55fd1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:14.918 INFO 2458 --- [ocp-schedules-12,f7df394c2333cc09,9e3b4015c222e278] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:14.921 INFO 2458 --- [ocp-schedules-12,f7df394c2333cc09,9e3b4015c222e278] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:15.351 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:15.354 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:15.413 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535380, nodeCount=1000, affectRows=1000 2025-10-27 11:24:15.417 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535380, nodeCount=999, validCount=987 2025-10-27 11:24:15.495 INFO 2458 --- [ocp-schedules-11,836afb379df5132c,183ea6d8810c4d2e] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:24:15.495+08:00, clusters=[1, 2] 2025-10-27 11:24:15.497 INFO 2458 --- [ocp-schedules-11,836afb379df5132c,183ea6d8810c4d2e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:24:15.498 INFO 2458 --- [ocp-schedules-11,836afb379df5132c,183ea6d8810c4d2e] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:24:16.575 WARN 2458 --- [ocp-async-2,afc391861f82af79,0753b32b81b3e65e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:16.575 INFO 2458 --- [ocp-async-2,0c609b4d43b45af5,3663a736b0d828c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:16.651 INFO 2458 --- [ocp-schedules-30,8918a28c45af2739,311ce324c3fbec85] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:16.816 INFO 2458 --- [ocp-schedules-6,ede2e461b53dd0fd,7e66ef79176997f7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:24:16.816 INFO 2458 --- [ocp-schedules-6,ede2e461b53dd0fd,7e66ef79176997f7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:24:16.873 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:24:16.874 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:24:16.874 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:24:16.874 INFO 2458 --- [Collect all tenants major compactions5,ede2e461b53dd0fd,446cbb68f0d5db5b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:24:16.874 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:24:16.875 INFO 2458 --- [Collect all tenants major compactions6,ede2e461b53dd0fd,2cd0ccf440c0c4ae] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:24:16.919 INFO 2458 --- [Collect all tenants major compactions5,ede2e461b53dd0fd,446cbb68f0d5db5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.920 INFO 2458 --- [Collect all tenants major compactions6,ede2e461b53dd0fd,2cd0ccf440c0c4ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.921 INFO 2458 --- [Collect all tenants major compactions5,ede2e461b53dd0fd,446cbb68f0d5db5b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:24:16.921 INFO 2458 --- [Collect all tenants major compactions6,ede2e461b53dd0fd,2cd0ccf440c0c4ae] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:24:16.930 INFO 2458 --- [Collect all tenants major compactions5,ede2e461b53dd0fd,446cbb68f0d5db5b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:24:16.932 INFO 2458 --- [Collect all tenants major compactions6,ede2e461b53dd0fd,2cd0ccf440c0c4ae] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:24:16.939 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.939 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.939 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.939 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.940 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.941 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:16.942 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:24:16.943 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:24:16.944 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:24:16.944 INFO 2458 --- [Collect all tenants major compactions1,ede2e461b53dd0fd,52e5ecf91497a1d0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:24:16.946 INFO 2458 --- [Collect all tenants major compactions3,ede2e461b53dd0fd,37712dcceee38c49] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:24:16.947 INFO 2458 --- [Collect all tenants major compactions4,ede2e461b53dd0fd,ae5e412cb8da367c] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:24:16.948 INFO 2458 --- [Collect all tenants major compactions2,ede2e461b53dd0fd,984a7a57783e8dee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:24:16.949 INFO 2458 --- [ocp-schedules-6,ede2e461b53dd0fd,7e66ef79176997f7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:24:17.707 INFO 2458 --- [prometheus-minute-schedule-1,815e3924a125687a,fff3d3abefe700f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:17.714 INFO 2458 --- [prometheus-minute-schedule-1,815e3924a125687a,fff3d3abefe700f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:18.354 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:18.355 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:18.525 INFO 2458 --- [ocp-async-5,4f3434347ad546ca,dd085786087bba21] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:18.575 WARN 2458 --- [ocp-async-8,7a7992efb0f3b291,34b5d01c45e53293] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:18.575 INFO 2458 --- [ocp-async-1,3ec07615cbfad136,2f250e394cc3480b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:19.857 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853} 2025-10-27 11:24:19.867 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo= 2025-10-27 11:24:19.868 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:24:19.870 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:19.873 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:24:19.874 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:19.876 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.878 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.878 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:24:19.878 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:19.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.882 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.882 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:24:19.882 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:19.884 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.891 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.891 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:24:19.891 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:19.897 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:24:19.906 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 1 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.708, end=2025-10-27T11:22:19.853},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:24:19.934 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:24:19.934 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:24:20.132 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449060132000, endTimeUs=1761535339853000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:24:20.132Z, endTime=2025-10-27T03:22:19.853Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:24:20.144 INFO 2458 --- [ocp-schedules-26,224dfb41687e929d,abcf0ef9422ae9b0] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:24:20.146 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:24:20.146 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:24:20.178 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:24:20.178 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535339853000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:24:20.184 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:24:20.184 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:24:20.208 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:24:20.208 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535339853000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:24:20.212 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:24:20.212 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:24:20.132, end=2025-10-27T11:22:19.853}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:24:20.213 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:24:20.434 INFO 2458 --- [ocp-schedules-4,4573fd4069174082,2b3a3cc8d8356a65] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:24:20.434 INFO 2458 --- [ocp-schedules-4,4573fd4069174082,2b3a3cc8d8356a65] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:24:20.443 INFO 2458 --- [ocp-schedules-4,4573fd4069174082,2b3a3cc8d8356a65] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:24:20.443 INFO 2458 --- [ocp-schedules-4,4573fd4069174082,2b3a3cc8d8356a65] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:24:20.575 WARN 2458 --- [ocp-async-7,47151343b99bfd7f,7ed042c40158fb38] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:20.575 INFO 2458 --- [ocp-async-10,71fb64c06befd862,7db96c989ce1e1e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:20.707 INFO 2458 --- [prometheus-minute-schedule-1,2dfa950818fe2139,842ed8c8fef5d8fd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:20.715 INFO 2458 --- [prometheus-minute-schedule-1,2dfa950818fe2139,842ed8c8fef5d8fd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:21.355 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:21.356 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:21.436 INFO 2458 --- [ocp-schedules-20,22f1f91e3d59284c,2ba6689674b29a1b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:21.436 INFO 2458 --- [ocp-schedules-6,e437b5e78c42a901,b14510033fa28cb6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:21.437 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:24:21.437 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:24:21.440 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:24:21.442 INFO 2458 --- [ocp-schedules-8,60b36330e4ac2932,cdca6215288accf9] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:24:21.456 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:21.457 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:21.457 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:21.458 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:21.458 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:21.458 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:21.459 INFO 2458 --- [ocp-schedules-26,e0a0a48bf5299452,74e0615ac46fdfd1] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-clockdiff1,e0a0a48bf5299452,54008802d8c78209] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-clockdiff2,e0a0a48bf5299452,56592633fb20be2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-clockdiff3,e0a0a48bf5299452,1a50922afad3532c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-clockdiff4,e0a0a48bf5299452,1e268b097e2cb77a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-clockdiff5,e0a0a48bf5299452,492f6087414aaa31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:24:21.460 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:21.461 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:21.462 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:21.462 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:21.463 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:21.463 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:21.463 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:21.463 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:21.463 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:21.464 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:21.485 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:21.465 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:21.510 INFO 2458 --- [host-check-clockdiff5,e0a0a48bf5299452,492f6087414aaa31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535461 0 0 , error= 2025-10-27 11:24:21.511 INFO 2458 --- [host-check-clockdiff3,e0a0a48bf5299452,1a50922afad3532c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535461 -1 -1 , error= 2025-10-27 11:24:21.522 INFO 2458 --- [host-check-clockdiff1,e0a0a48bf5299452,54008802d8c78209] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535461 0 0 , error= 2025-10-27 11:24:21.522 INFO 2458 --- [host-check-clockdiff4,e0a0a48bf5299452,1e268b097e2cb77a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535461 -1 -1 , error= 2025-10-27 11:24:21.530 INFO 2458 --- [host-check-clockdiff2,e0a0a48bf5299452,56592633fb20be2b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535461 -1 -1 , error= 2025-10-27 11:24:21.545 INFO 2458 --- [ocp-schedules-2,ae7ac3da705187b8,5d08dacf87d74b91] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:24:21.549 INFO 2458 --- [ocp-schedules-16,f81a96ffa4909752,64e67b3d5421e0cb] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:24:21.554 INFO 2458 --- [ocp-schedules-16,f81a96ffa4909752,64e67b3d5421e0cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:21.554 INFO 2458 --- [ocp-schedules-18,76ce467d7a21ceaa,c70a2f78b3872f1b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:24:21.555 INFO 2458 --- [ocp-schedules-2,ae7ac3da705187b8,5d08dacf87d74b91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:21.557 INFO 2458 --- [ocp-schedules-16,f81a96ffa4909752,64e67b3d5421e0cb] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:24:21.559 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:24:21.560 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:24:21.560 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:24:21.561 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:24:21.561144362+08:00 2025-10-27 11:24:21.562 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:24:21.561+08:00 2025-10-27 11:24:21.562 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:24:21.562138424+08:00 2025-10-27 11:24:21.562 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:24:21.562 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:24:21.562 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:24:21.562+08:00 2025-10-27 11:24:21.563 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:24:21.562330172+08:00 2025-10-27 11:24:21.564 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:24:21.563+08:00 2025-10-27 11:24:21.564 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:24:21.563069200+08:00 2025-10-27 11:24:21.565 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:24:21.565 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:24:21.565 INFO 2458 --- [host-check-agent-status4,22f1f91e3d59284c,23e0179026418713] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:24:21.565 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:24:21.564+08:00 2025-10-27 11:24:21.565 INFO 2458 --- [ocp-schedules-18,76ce467d7a21ceaa,c70a2f78b3872f1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:21.566 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:24:21.568 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:24:21.568 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:24:21.568 INFO 2458 --- [host-check-agent-status5,22f1f91e3d59284c,3863e411339b99c2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:24:21.570 INFO 2458 --- [ocp-schedules-4,250536c4fd89d1fa,648c62e99eeb320f] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:24:21.572 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:24:21.561+08:00 and status AVAILABLE 2025-10-27 11:24:21.573 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:24:21.562+08:00 and status AVAILABLE 2025-10-27 11:24:21.573 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:24:21.563+08:00 and status AVAILABLE 2025-10-27 11:24:21.576 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:24:21.577 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:24:21.577 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:24:21.578 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:24:21.577536662+08:00 2025-10-27 11:24:21.579 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:24:21.578+08:00 2025-10-27 11:24:21.579 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:24:21.564+08:00 and status AVAILABLE 2025-10-27 11:24:21.580 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:24:21.580 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:24:21.581 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:24:21.581 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:24:21.581 INFO 2458 --- [host-check-agent-status3,22f1f91e3d59284c,1e40c19e78fbec35] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:24:21.587 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:24:21.590 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:24:21.578+08:00 and status AVAILABLE 2025-10-27 11:24:21.590 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:24:21.594 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:24:21.594 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:24:21.594 INFO 2458 --- [host-check-agent-status1,22f1f91e3d59284c,ac91b613bb36b066] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:24:21.595 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:24:21.604 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:21.605 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:21.607 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:21.607 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:21.608 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:21.610 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:21.611 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:21.614 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:21.614 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:21.616 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:24:21.617 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:21.617 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:24:21.621 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:21.621 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:24:21.621 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:24:21.621 INFO 2458 --- [host-check-agent-status2,22f1f91e3d59284c,fa64d83a2502792e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:24:21.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.631 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.631 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.631 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.632 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.632 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:21.637 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:24:21.637 INFO 2458 --- [ocp-schedules-2,ae7ac3da705187b8,5d08dacf87d74b91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:24:21.659 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.660 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:21.675 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:21.686 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:21.691 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:21.697 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:21.699 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:21.706 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.706 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.706 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.706 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.707 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.707 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:24:21.707 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:21.709 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:24:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:21.715136+08:00, lastProcessTime=2025-10-27T11:24:11.714856+08:00 2025-10-27 11:24:21.720 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:24:21.720 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:24:21.733 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:24:21.742 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:21.743 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.743 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:24:21.748 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:24:21.750 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:24:21.753 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:24:21.755 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:24:21.766 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:24:21.783 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:21.787 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:21.789 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:21.793 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.794 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.794 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.794 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.795 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:24:21.796 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:21.801 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.802 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:21.802 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:24:21.803 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.806 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:24:21.806 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:21.807 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:24:21.810 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:24:21.812 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:24:21.830 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:21.830 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:21.831 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:24:21.836 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:24:21.839 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:24:21.839 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:24:21.843 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:24:21.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:21.859 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:24:21.866 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:21.867 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:21.870 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:21.872 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:21.881 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:24:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:21.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:21.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:21.884 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:21.884 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.885 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:21.886 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:24:21.886 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:24:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:21.887 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:24:21.890 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:24:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:21.892 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:24:21.897 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:21.898 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:24:21.914 INFO 2458 --- [host-check-host-status4,e437b5e78c42a901,3eca4b5d469ff94e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:24:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:21.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:21.930 INFO 2458 --- [host-check-host-status3,e437b5e78c42a901,998d3583b95679ae] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:24:21.932 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:24:21.932 INFO 2458 --- [host-check-host-status5,e437b5e78c42a901,e4223efe91fa3419] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:24:21.938 INFO 2458 --- [host-check-host-status2,e437b5e78c42a901,36a040369d4a130a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:24:21.949 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:21.967 INFO 2458 --- [host-check-host-status1,e437b5e78c42a901,41f327bba3191fdf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:24:21.991 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:21.715136+08:00 2025-10-27 11:24:21.992 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:24:21.994 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:21.994 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:22.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:22.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:22.034 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:24:22.035 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.035 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:24:22.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.038 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:24:22.056 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:24:22.072 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:22.072 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:22.078 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.079 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:24:22.104 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:24:22.104 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:24:22.106 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:22.106 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:22.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.183 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.183 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.183 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.183 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:22.187 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:24:22.197 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:24:22.206 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:22.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:22.254 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:24:22.304 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:24:22.304 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.304 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.306 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.307 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:24:22.308 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.308 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.309 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.309 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:24:22.348 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:24:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:24:22.434 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:24:22.434 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:24:22.434 INFO 2458 --- [ocp-schedules-18,76ce467d7a21ceaa,c70a2f78b3872f1b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:24:22.576 WARN 2458 --- [ocp-async-3,745a9c0094199cc1,37e0aa5a38e8d632] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:22.576 INFO 2458 --- [ocp-async-3,ba925d5771464ea3,e07d9e2986e8e635] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:23.521 INFO 2458 --- [ocp-schedules-10,db241d33d829f2e4,207d7ea64802d986] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:23.552 WARN 2458 --- [scanner-112183215-1,db241d33d829f2e4,4a01e5d5d13e3758] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:23.707 INFO 2458 --- [prometheus-minute-schedule-1,7f6a461087ef5bcb,a3f4f5ceee0d5d2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:23.714 INFO 2458 --- [prometheus-minute-schedule-1,7f6a461087ef5bcb,a3f4f5ceee0d5d2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:24.356 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:24.358 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:24.456 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,e48be625016443fe,1202483076c699a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=e48be625016443fe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:24:24.456 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,e48be625016443fe,1202483076c699a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=e48be625016443fe, duration=0 ms] 2025-10-27 11:24:24.576 INFO 2458 --- [ocp-async-5,7725df5e682c4d22,9ce98cec33898cbe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:24.576 WARN 2458 --- [ocp-async-6,aafe9d8c6865a950,334c9a61052b9355] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:24.929 INFO 2458 --- [ocp-schedules-29,e4ea47097c497c51,f16886254bf80dba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:24.933 INFO 2458 --- [ocp-schedules-29,e4ea47097c497c51,f16886254bf80dba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:26.576 WARN 2458 --- [ocp-async-1,21df5f77aa42331c,0d935960f3db0a7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:26.576 INFO 2458 --- [ocp-async-7,672e88e461aa2ca8,46fc4b8bef9a16ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:26.660 INFO 2458 --- [ocp-schedules-24,b2f325de6736dbf6,474bf5f7a9ee89b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:26.707 INFO 2458 --- [prometheus-minute-schedule-1,68651a8ddf09325e,e91643d8734ae932] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:26.713 INFO 2458 --- [prometheus-minute-schedule-1,68651a8ddf09325e,e91643d8734ae932] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:27.332 INFO 2458 --- [ocp-schedules-29,3c317a58a86e2941,7b9e17d296f48817] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:24:27.345 INFO 2458 --- [ocp-schedules-29,3c317a58a86e2941,7b9e17d296f48817] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:27.345 INFO 2458 --- [ocp-schedules-29,3c317a58a86e2941,7b9e17d296f48817] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:27.358 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:27.359 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:28.531 INFO 2458 --- [ocp-async-10,19f69638edc3e11a,879030e1b34a163a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:28.576 INFO 2458 --- [ocp-async-2,d07ab8908d241bbe,82db28fdf09dfd69] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:28.576 WARN 2458 --- [ocp-async-3,0861219b310fe9c3,5a2e165323a177fc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:28.976 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:24:28.976 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:24:28.981 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:24:28.981 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:24:28.987 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:24:28.987 INFO 2458 --- [ocp-schedules-15,b0ceb832a2d4f481,fbeff77cb24b5c09] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:24:29.442 INFO 2458 --- [ocp-schedules-24,4c60795a3d47ba12,cbfce5303340e4fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:24:29.449 INFO 2458 --- [ocp-schedules-24,4c60795a3d47ba12,cbfce5303340e4fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:24:29.449 INFO 2458 --- [ocp-schedules-24,4c60795a3d47ba12,cbfce5303340e4fb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:24:29.707 INFO 2458 --- [prometheus-minute-schedule-1,3287c38be5c08e2f,721b7f50fc1171a0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:29.715 INFO 2458 --- [prometheus-minute-schedule-1,3287c38be5c08e2f,721b7f50fc1171a0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:30.359 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:30.360 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:30.576 WARN 2458 --- [ocp-async-5,7cc40f3ba5099b89,bb8274af3091111a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:30.576 INFO 2458 --- [ocp-async-6,8a8899388f81f7e8,2450b890cd8b0450] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:31.716267+08:00, lastProcessTime=2025-10-27T11:24:21.715136+08:00 2025-10-27 11:24:31.922 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:31.716267+08:00 2025-10-27 11:24:32.576 INFO 2458 --- [ocp-async-1,e903a5f7ff03645e,654d55c2db322f30] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:32.576 WARN 2458 --- [ocp-async-7,af1adb15a07fae8a,58700cb7f91caea5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:32.707 INFO 2458 --- [prometheus-minute-schedule-1,f341271eacf4ac9b,a29d003aaaec0236] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:32.715 INFO 2458 --- [prometheus-minute-schedule-1,f341271eacf4ac9b,a29d003aaaec0236] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:33.360 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:33.361 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:33.645 INFO 2458 --- [ocp-schedules-20,5a0fb18287e0f69e,bb9d2e0c302b9e57] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:33.697 WARN 2458 --- [scanner-112183388-1,5a0fb18287e0f69e,b70f475f038c3393] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:33.820 INFO 2458 --- [alarm-detect-12,5a0fb18287e0f69e,ec1130de40969d61] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535440 ~ 1761535463 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535455, value=52.951)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535455, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535455=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:24:33.895 INFO 2458 --- [ocp-schedules-18,7283e3d3cb1683e4,efd3ea8599932d0c] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:24:33.895 INFO 2458 --- [ocp-schedules-18,7283e3d3cb1683e4,efd3ea8599932d0c] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:24:34.293 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:24:34.293 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:24:34.314 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:24:34.315 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:24:34.315 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:24:34.329 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:24:34.329 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:24:34.329 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:24:34.355 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:24:34.355 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:24:34.355 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:24:34.372 INFO 2458 --- [ocp-schedules-27,9941982a66725b41,56f6f2776335acd8] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:24:34.576 INFO 2458 --- [ocp-async-10,c6655f9156ea19cc,6569ac322f26e84e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:34.576 WARN 2458 --- [ocp-async-9,863a7875be3fc07c,c4a795f01cb74f2e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:34.939 INFO 2458 --- [ocp-schedules-29,0bdfe19bba6425d6,79b9e316d170d126] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:34.944 INFO 2458 --- [ocp-schedules-29,0bdfe19bba6425d6,79b9e316d170d126] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:35.707 INFO 2458 --- [prometheus-minute-schedule-1,3c8972bbb45bfb9a,2805468820a0b746] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:35.716 INFO 2458 --- [prometheus-minute-schedule-1,3c8972bbb45bfb9a,2805468820a0b746] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:36.362 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:36.363 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:36.577 INFO 2458 --- [ocp-async-5,78a313589be36149,75df569ca58a7317] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:36.577 WARN 2458 --- [ocp-async-8,fc0a78bd44cf5143,c38a60aada82a3de] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:36.667 INFO 2458 --- [ocp-schedules-12,78f4036ace560efc,924cf0cb8635fb1c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:37.708 INFO 2458 --- [prometheus-cache-monitor1,bc1747f23e16230a,ac34ffcca458eafe] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20389,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5096,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:24:37.708 INFO 2458 --- [prometheus-cache-monitor1,bc1747f23e16230a,ac34ffcca458eafe] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:24:38.364 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:24:38.365 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:24:38.433 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:38.433 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:38.477 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:38.477 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:38.477 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:38.478 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:24:38.490 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:38.490 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:38.490 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:38.491 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:24:38.531 INFO 2458 --- [ocp-async-5,69549edc679b5aff,7d2d7952190e0ea7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:38.552 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:38.553 INFO 2458 --- [ocp-schedules-16,e1154d4f53486967,fcb5a89c5fc8badc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:24:38.577 WARN 2458 --- [ocp-async-1,762e758c7971582f,788795cd6b7ea6e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:38.577 INFO 2458 --- [ocp-async-4,1cd076d3288a423e,7b75e87b2465e804] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:38.707 INFO 2458 --- [prometheus-minute-schedule-1,3fca07d819e98528,40355cc87d63ac93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:38.714 INFO 2458 --- [prometheus-minute-schedule-1,3fca07d819e98528,40355cc87d63ac93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:38.718 INFO 2458 --- [ocp-schedules-19,0b066c84446c3080,86739f1308451b69] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:24:38.725 INFO 2458 --- [ocp-schedules-19,0b066c84446c3080,86739f1308451b69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:38.725 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:24:38.725 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:24:38.725 INFO 2458 --- [ocp-schedules-19,0b066c84446c3080,86739f1308451b69] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:24:38.993 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:38.993 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:24:38.993 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:24:38.993 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:24:38.994 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:24:38.994 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:24:39.004 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:39.005 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:39.008 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:39.009 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:39.009 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:39.010 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:39.010 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:39.012 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:39.012 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:39.013 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:39.067 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:24:39.073 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:24:39.073 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:24:39.074 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:24:39.085 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:24:39.087 INFO 2458 --- [ocp-schedules-18,7098c89e1722cf7e,19a0eed99d0324b1] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:24:39.363 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:39.364 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:40.013 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:24:40.013 INFO 2458 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:24:40.052 INFO 2458 --- [Check obproxy1,7265077287455347,c9e3ad04f277531d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:24:40.052 INFO 2458 --- [Check obproxy1,7265077287455347,c9e3ad04f277531d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:24:40.052 INFO 2458 --- [Collect obproxy1,953e7e4e7428d461,08070dc9ff148a67] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:24:40.052 INFO 2458 --- [Collect obproxy1,953e7e4e7428d461,08070dc9ff148a67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:24:40.081 INFO 2458 --- [Collect obproxy1,953e7e4e7428d461,08070dc9ff148a67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:24:40.084 INFO 2458 --- [Collect obproxy1,953e7e4e7428d461,08070dc9ff148a67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:24:40.085 INFO 2458 --- [Check obproxy1,7265077287455347,c9e3ad04f277531d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:24:40.088 INFO 2458 --- [Collect obproxy1,953e7e4e7428d461,08070dc9ff148a67] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:24:40.088 INFO 2458 --- [Check obproxy1,7265077287455347,c9e3ad04f277531d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:24:40.088 INFO 2458 --- [Check obproxy1,7265077287455347,c9e3ad04f277531d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:24:40.577 INFO 2458 --- [ocp-async-9,8ffec3932d5b3b4a,6024f14b06cd388d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:40.577 WARN 2458 --- [ocp-async-10,355d5baad1da93cd,f6efd795a66cf836] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:41.168 INFO 2458 --- [ocp-schedules-23,d3d6703d19721125,30cadc84f7fcd6d3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:24:41.169 INFO 2458 --- [ocp-schedules-23,d3d6703d19721125,30cadc84f7fcd6d3] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:24:41.707 INFO 2458 --- [prometheus-minute-schedule-1,8d3cb35c28a296bd,7cbd8e28c336b004] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:41.714 INFO 2458 --- [prometheus-minute-schedule-1,8d3cb35c28a296bd,7cbd8e28c336b004] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:41.713859+08:00, lastProcessTime=2025-10-27T11:24:31.716267+08:00 2025-10-27 11:24:41.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:41.713859+08:00 2025-10-27 11:24:42.364 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:42.366 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:42.577 INFO 2458 --- [ocp-async-2,cc351604d8b79dc4,cb91b000b59b2813] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:42.577 WARN 2458 --- [ocp-async-6,8dd08b57130d8317,7d152219f9f0b4d5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:43.838 INFO 2458 --- [ocp-schedules-15,10211ae6191e92f9,87494a60d8a2735e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:43.860 WARN 2458 --- [scanner-112183615-1,10211ae6191e92f9,e239ad8479728701] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:43.955 INFO 2458 --- [alarm-detect-11,10211ae6191e92f9,fd3314f6a8ac401a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535455 ~ 1761535473 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535470, value=46.009)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535470, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535470=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:24:44.188 INFO 2458 --- [ocp-schedules-23,3e67f8eec3e9dfc9,2f02579478e7159f] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:24:44.577 INFO 2458 --- [ocp-async-1,722887d3b0a46bd0,0f19f0c6f2082cc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:44.577 WARN 2458 --- [ocp-async-7,77958b2058e9efc1,74e07db8e70ee6d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:44.707 INFO 2458 --- [prometheus-minute-schedule-1,0fecdd3b87dbba9e,31e095eaaf9f8b9d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:44.715 INFO 2458 --- [prometheus-minute-schedule-1,0fecdd3b87dbba9e,31e095eaaf9f8b9d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:44.951 INFO 2458 --- [ocp-schedules-8,aa781a4d494db50f,9393d7c580da9c6d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:44.954 INFO 2458 --- [ocp-schedules-8,aa781a4d494db50f,9393d7c580da9c6d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:45.102 INFO 2458 --- [prometheus-second-schedule-1,93999ea5cc6c7878,34c0d0c913832287] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:24:45.366 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:45.367 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:46.539 INFO 2458 --- [ocp-schedules-6,2cb806aa2a1965e5,66dfa3ec8ef4d4bf] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:24:46.539 INFO 2458 --- [ocp-schedules-6,2cb806aa2a1965e5,66dfa3ec8ef4d4bf] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:24:46.577 INFO 2458 --- [ocp-async-10,8f28cbd549ebfaac,a260a19416120e03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:46.577 WARN 2458 --- [ocp-async-3,4cd3885a96db6255,f338343e1786baf5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:46.607 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:24:46.607 INFO 2458 --- [pool-880-thread-2,2cb806aa2a1965e5,73a97ff87b76d5a2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:24:46.653 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:46.653 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:46.658 INFO 2458 --- [pool-880-thread-2,2cb806aa2a1965e5,73a97ff87b76d5a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:46.660 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:46.660 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:46.662 INFO 2458 --- [pool-880-thread-2,2cb806aa2a1965e5,73a97ff87b76d5a2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535186000000, 1761535486000000] 2025-10-27 11:24:46.662 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:46.665 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535186000000, 1761535486000000] 2025-10-27 11:24:46.674 INFO 2458 --- [ocp-schedules-4,86c4a4c624ff8870,a3a6796ee4d3e929] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:46.705 WARN 2458 --- [pool-880-thread-2,2cb806aa2a1965e5,73a97ff87b76d5a2] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:24:46.722 INFO 2458 --- [pool-880-thread-2,2cb806aa2a1965e5,73a97ff87b76d5a2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:24:46.764 WARN 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:24:46.764 WARN 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:24:46.764 WARN 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:24:46.772 INFO 2458 --- [pool-880-thread-1,2cb806aa2a1965e5,af703211962d621e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:24:46.773 INFO 2458 --- [ocp-schedules-6,2cb806aa2a1965e5,66dfa3ec8ef4d4bf] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:24:47.348 INFO 2458 --- [ocp-schedules-28,1c59e46563623b32,17988d5e563cd5fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:24:47.357 INFO 2458 --- [ocp-schedules-28,1c59e46563623b32,17988d5e563cd5fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:47.357 INFO 2458 --- [ocp-schedules-28,1c59e46563623b32,17988d5e563cd5fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:24:47.707 INFO 2458 --- [prometheus-minute-schedule-1,c03e3697e1e19ef2,716a89c4b2bca0f3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:47.715 INFO 2458 --- [prometheus-minute-schedule-1,c03e3697e1e19ef2,716a89c4b2bca0f3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:48.367 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:48.368 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:48.532 INFO 2458 --- [ocp-async-10,042c0b1e36f0bb80,42ca967eaf232312] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:48.578 INFO 2458 --- [ocp-async-2,fa16844537275b01,68fa40a0c9f164db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:48.578 WARN 2458 --- [ocp-async-8,3e4e0b9fcd3020d2,db196e3c2101e66b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:49.195 INFO 2458 --- [ocp-schedules-16,e58b25c1d4c3dacf,08c16f7ac0324c89] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:24:49.195+08:00, clusters=[1, 2] 2025-10-27 11:24:49.210 INFO 2458 --- [ocp-schedules-16,e58b25c1d4c3dacf,08c16f7ac0324c89] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:24:49.213 INFO 2458 --- [ocp-schedules-16,e58b25c1d4c3dacf,08c16f7ac0324c89] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:24:50.578 WARN 2458 --- [ocp-async-5,c604fa5b73cf6f68,923125a9625d6ca8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:50.578 INFO 2458 --- [ocp-async-5,f0346c02ac6c8d90,f8a55808f5034607] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:50.590 INFO 2458 --- [ocp-schedules-15,fa38f8bcdd3651ff,3456b36dea505940] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:24:50.590 INFO 2458 --- [ocp-schedules-15,fa38f8bcdd3651ff,3456b36dea505940] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:24:50.597 INFO 2458 --- [ocp-schedules-15,fa38f8bcdd3651ff,3456b36dea505940] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:24:50.597 INFO 2458 --- [ocp-schedules-15,fa38f8bcdd3651ff,3456b36dea505940] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:24:50.707 INFO 2458 --- [prometheus-minute-schedule-1,76a4bf889a2bb455,62979c7d638e7a2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:50.716 INFO 2458 --- [prometheus-minute-schedule-1,76a4bf889a2bb455,62979c7d638e7a2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:51.368 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:51.370 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:51.409 INFO 2458 --- [ocp-schedules-24,ae825108b43f356b,0b776909f7d4aecf] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:51.409 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:24:51.410 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:24:51.410 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:24:51.410 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:24:51.410 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:24:51.411 INFO 2458 --- [ocp-schedules-5,c5ebaa2e79494832,d5f010c57d2f240b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:51.411 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:24:51.411 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:24:51.411 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:24:51.411 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:24:51.414 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:24:51.424 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:51.424 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:51.424 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:51.425 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:51.426 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:51.427 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:51.428 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:51.429 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:51.429 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:51.430 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:51.430 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:51.430 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:51.430 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:51.431 INFO 2458 --- [ocp-schedules-21,e8baf2df92bc46ab,d5ab48849d4c26ae] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:24:51.431 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:51.431 INFO 2458 --- [host-check-clockdiff1,e8baf2df92bc46ab,1c896469cd6ab50f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:24:51.431 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:51.432 INFO 2458 --- [host-check-clockdiff2,e8baf2df92bc46ab,8db5d7083380bd22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:24:51.433 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:51.433 INFO 2458 --- [host-check-clockdiff3,e8baf2df92bc46ab,962420852c522f01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:24:51.433 INFO 2458 --- [host-check-clockdiff4,e8baf2df92bc46ab,c4383e05f938be15] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:24:51.434 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:51.434 INFO 2458 --- [host-check-clockdiff5,e8baf2df92bc46ab,0ac13850bcc1e298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:24:51.437 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:51.437 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:51.437 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:51.451 INFO 2458 --- [ocp-schedules-29,059cc458a7a0115c,93f692587053b9de] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:24:51.474 INFO 2458 --- [host-check-clockdiff5,e8baf2df92bc46ab,0ac13850bcc1e298] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535491 0 0 , error= 2025-10-27 11:24:51.477 INFO 2458 --- [host-check-clockdiff2,e8baf2df92bc46ab,8db5d7083380bd22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535491 -1 -1 , error= 2025-10-27 11:24:51.478 INFO 2458 --- [host-check-clockdiff1,e8baf2df92bc46ab,1c896469cd6ab50f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535491 0 0 , error= 2025-10-27 11:24:51.483 INFO 2458 --- [host-check-clockdiff3,e8baf2df92bc46ab,962420852c522f01] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535491 -1 -1 , error= 2025-10-27 11:24:51.483 INFO 2458 --- [host-check-clockdiff4,e8baf2df92bc46ab,c4383e05f938be15] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535491 0 0 , error= 2025-10-27 11:24:51.517 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:24:51.517 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:24:51.518 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:24:51.518 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:24:51.518643481+08:00 2025-10-27 11:24:51.519 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:24:51.519 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:24:51.518+08:00 2025-10-27 11:24:51.520 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:24:51.519851688+08:00 2025-10-27 11:24:51.520 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-agent-status5,c5ebaa2e79494832,b974874f3ab4bd29] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:24:51.519773708+08:00 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:24:51.520+08:00 2025-10-27 11:24:51.521 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:24:51.521+08:00 2025-10-27 11:24:51.522 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:24:51.521867059+08:00 2025-10-27 11:24:51.523 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:24:51.522+08:00 2025-10-27 11:24:51.523 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:24:51.523 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:24:51.525 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:24:51.524996057+08:00 2025-10-27 11:24:51.525 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:24:51.526 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:24:51.525+08:00 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status4,c5ebaa2e79494832,61c776b07898b6e0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:24:51.527 INFO 2458 --- [host-check-agent-status1,c5ebaa2e79494832,1b9bd5fe18c52824] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:24:51.528 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:24:51.518+08:00 and status AVAILABLE 2025-10-27 11:24:51.530 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:24:51.521+08:00 and status AVAILABLE 2025-10-27 11:24:51.530 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:24:51.520+08:00 and status AVAILABLE 2025-10-27 11:24:51.531 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:24:51.522+08:00 and status AVAILABLE 2025-10-27 11:24:51.531 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:24:51.533 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:24:51.533 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:24:51.525+08:00 and status AVAILABLE 2025-10-27 11:24:51.533 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:24:51.533 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:24:51.533 INFO 2458 --- [host-check-agent-status3,c5ebaa2e79494832,e7a8ac7f61c23572] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:24:51.534 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:24:51.536 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:24:51.536 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:24:51.538 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:24:51.539 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:24:51.542 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:24:51.542 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:24:51.542 INFO 2458 --- [host-check-agent-status2,c5ebaa2e79494832,6880d1d2b29ab2d2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:24:51.545 INFO 2458 --- [ocp-schedules-18,9e42758124a1b401,5a1448b4f50c12d3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:24:51.549 INFO 2458 --- [ocp-schedules-11,301fd7f5386bf837,a469acf0159648f2] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:24:51.554 INFO 2458 --- [ocp-schedules-11,301fd7f5386bf837,a469acf0159648f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:51.554 INFO 2458 --- [ocp-schedules-22,be46a23309bdf250,71dedfcad1caa18b] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:24:51.554 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:51.554 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:51.554 INFO 2458 --- [ocp-schedules-11,301fd7f5386bf837,a469acf0159648f2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:24:51.556 INFO 2458 --- [ocp-schedules-18,9e42758124a1b401,5a1448b4f50c12d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:51.557 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:24:51.557 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:51.557 INFO 2458 --- [ocp-schedules-22,be46a23309bdf250,71dedfcad1caa18b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:24:51.558 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:51.559 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:24:51.560 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:24:51.562 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:24:51.562 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:24:51.564 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:24:51.566 INFO 2458 --- [ocp-schedules-13,da6b822e120acd74,b6586432dda51c86] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:24:51.568 INFO 2458 --- [ocp-schedules-25,20a710e7156d1d36,636aecb1b46bde9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:24:51.569 INFO 2458 --- [ocp-schedules-30,78c3b19868d67996,df0d75a5b5f36d65] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:24:51.583 INFO 2458 --- [ocp-schedules-30,78c3b19868d67996,df0d75a5b5f36d65] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:24:51.588 INFO 2458 --- [ocp-schedules-25,20a710e7156d1d36,636aecb1b46bde9d] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:24:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:24:51.623 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:51.624 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:51.627 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:51.631 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:51.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:24:51.635 INFO 2458 --- [ocp-schedules-18,9e42758124a1b401,5a1448b4f50c12d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:24:51.636 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:24:51.643 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.644 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:51.646 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:24:51.647 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:24:51.649 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:24:51.651 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:24:51.658 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:24:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.688 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.689 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:51.692 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:24:51.694 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:24:51.694 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:24:51.701 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:24:51.707 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:24:51.714 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:24:51.713788+08:00, lastProcessTime=2025-10-27T11:24:41.713859+08:00 2025-10-27 11:24:51.715 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.716 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:24:51.730 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:51.733 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:51.734 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:51.734 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:51.740 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:24:51.751 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:24:51.752 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:24:51.753 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:24:51.755 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:24:51.770 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:24:51.773 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:51.773 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.773 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.774 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.774 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.774 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.774 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:24:51.775 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:51.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:51.777 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:24:51.781 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:51.782 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:24:51.783 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:24:51.790 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:24:51.801 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:24:51.804 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:51.804 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:51.811 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:24:51.813 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:51.817 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:51.829 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:51.830 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:51.830 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:51.834 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:24:51.834 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:24:51.841 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:24:51.845 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:24:51.850 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:24:51.857 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:51.858 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:51.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.871 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:24:51.871 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:24:51.871 INFO 2458 --- [host-check-host-status2,ae825108b43f356b,9c2a86a3f240f85f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:24:51.873 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:51.873 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:51.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:24:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:51.879 INFO 2458 --- [host-check-host-status3,ae825108b43f356b,c0c0611d4c85eada] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:24:51.880 INFO 2458 --- [host-check-host-status5,ae825108b43f356b,3443b2d08e27e11d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:24:51.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:24:51.882 INFO 2458 --- [host-check-host-status4,ae825108b43f356b,131062e5668163c2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:24:51.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:24:51.901 INFO 2458 --- [host-check-host-status1,ae825108b43f356b,53ed53d6c3eba061] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:24:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:51.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:24:51.983 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:24:51.984 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.984 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:24:51.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:51.986 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:51.986 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:51.987 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:24:52.002 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:24:51.713788+08:00 2025-10-27 11:24:52.004 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:24:52.022 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:52.022 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:52.027 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:52.027 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:52.028 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:24:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:24:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:24:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:24:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:24:52.140 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:24:52.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:52.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.182 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.183 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.183 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.183 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:52.184 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:24:52.187 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:24:52.198 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:24:52.208 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:24:52.212 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:24:52.260 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:24:52.309 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:24:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:52.310 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:24:52.311 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.312 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.312 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:52.313 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:24:52.348 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:24:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:24:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:24:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:24:52.404 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:24:52.404 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:24:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:24:52.444 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:24:52.444 INFO 2458 --- [ocp-schedules-22,be46a23309bdf250,71dedfcad1caa18b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:24:52.578 WARN 2458 --- [ocp-async-9,d3d10346982c5ae1,b2ddc33cc2b3c9d1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:52.578 INFO 2458 --- [ocp-async-8,aac1d829e18e6494,ea8583d98d55a53b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:24:53.707 INFO 2458 --- [prometheus-minute-schedule-1,130c2ab8f0824817,a7fc848f9779f328] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:53.722 INFO 2458 --- [prometheus-minute-schedule-1,130c2ab8f0824817,a7fc848f9779f328] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:53.961 INFO 2458 --- [ocp-schedules-12,1ee3705a0e0c2a23,1c09a8665b267b46] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:24:53.985 WARN 2458 --- [scanner-112183814-1,1ee3705a0e0c2a23,dead56ad1c763bbd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:24:54.085 INFO 2458 --- [alarm-detect-13,1ee3705a0e0c2a23,bdf2d9beb68d6aa9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535424 ~ 1761535484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535440, value=93.131)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.131, values={1761535440=93.131}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:24:54.094 INFO 2458 --- [alarm-detect-15,1ee3705a0e0c2a23,0682bd2c0d2258cb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535424 ~ 1761535484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535440, value=203774.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203774.56, values={1761535440=203774.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:24:54.127 INFO 2458 --- [alarm-detect-14,1ee3705a0e0c2a23,e5af4de907b79121] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535424 ~ 1761535484 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535440, value=203775.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203775.912, values={1761535440=203775.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:24:54.370 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:54.371 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:54.578 WARN 2458 --- [ocp-async-6,a801b6c7afaa4515,f392025b9f4b33d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:54.578 INFO 2458 --- [ocp-async-1,dd01d9ba6fbf19c5,e910831b323c81a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:54.962 INFO 2458 --- [ocp-schedules-20,1ebf79dbf3e8b84e,09267d91c7ca2fe3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:24:54.966 INFO 2458 --- [ocp-schedules-20,1ebf79dbf3e8b84e,09267d91c7ca2fe3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:24:55.233 INFO 2458 --- [ocp-schedules-25,c1f3e10fb0dacf3b,47b7be50945eee48] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:24:56.578 INFO 2458 --- [ocp-async-7,7f479f301213f677,82d21ee9495535aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:56.578 WARN 2458 --- [ocp-async-4,ded8c8e5bf8f4961,b21bcaf84b060021] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:56.681 INFO 2458 --- [ocp-schedules-23,6ee45b19b64fe364,5cd9ef1e2027f89b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:24:56.707 INFO 2458 --- [prometheus-minute-schedule-1,819add2ff956fdc2,3147e22c9703ccc9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:56.713 INFO 2458 --- [prometheus-minute-schedule-1,819add2ff956fdc2,3147e22c9703ccc9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:24:57.371 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:24:57.372 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:24:58.532 INFO 2458 --- [ocp-async-7,728a5cfc97631472,ff59874bafa32c1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:24:58.578 INFO 2458 --- [ocp-async-5,f2f26cfb155a13a0,65d8725de415de18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:24:58.578 WARN 2458 --- [ocp-async-9,b4534231c4c87f88,ea4d83a6a7113bdb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:24:59.420 INFO 2458 --- [ocp-schedules-27,61297c9479e200a4,10f25d6ffc4dd33a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:24:59.608 INFO 2458 --- [ocp-schedules-30,a8dde07366799297,209617562a9e4e70] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:24:59.612 INFO 2458 --- [ocp-schedules-30,a8dde07366799297,209617562a9e4e70] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:24:59.707 INFO 2458 --- [prometheus-minute-schedule-1,458697fb2c175123,ead84ffb501198a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535500, interval=60 2025-10-27 11:24:59.715 INFO 2458 --- [prometheus-minute-schedule-1,458697fb2c175123,ead84ffb501198a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:00.372 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:00.373 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:00.578 INFO 2458 --- [ocp-async-6,9627627d4dc43e78,7d513229e5b9c20f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:00.578 WARN 2458 --- [ocp-async-5,dc0d8c8a2e0fae4b,548b70d05c44325b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:00.777 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:20:01.0, nextRun=2025-10-27 11:25:00.0, currentDate=Mon Oct 27 11:25:00 CST 2025 2025-10-27 11:25:00.777 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:20:01.0, nextRunTime=2025-10-27 11:25:00.0} 2025-10-27 11:25:00.796 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:25:00.796 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:25:00.796+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:25:00.838 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:20:01.0, nextRun=2025-10-27 11:25:00.0, currentDate=Mon Oct 27 11:25:00 CST 2025 2025-10-27 11:25:00.838 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:20:01.0, nextRunTime=2025-10-27 11:25:00.0} 2025-10-27 11:25:00.896 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:25:00.919 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:25:00.927 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:25:00.927 INFO 2458 --- [task-coordinator2,8964d58b78e8b300,f0893981f4506a55] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:25:00.927+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:25:01.495 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,0fde2ead51e05c42] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9163, operation:EXECUTE, state: RUNNING 2025-10-27 11:25:01.531 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,0fde2ead51e05c42] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9164, operation:EXECUTE, state: RUNNING 2025-10-27 11:25:01.562 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:25:01.563 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:25:01.566 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9163, context=Context{parallelIdx=-1, stringMap={task_instance_id=9105, task_operation=execute, latest_execution_start_time=2025-10-27T11:25:01.483+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9163}, listMap={}}, executor=10.40.65.146 2025-10-27 11:25:01.567 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9164, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9106, task_operation=execute, latest_execution_start_time=2025-10-27T11:25:01.528+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9164}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:25:01.567 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:25:01.623 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:01.714557+08:00, lastProcessTime=2025-10-27T11:24:51.713788+08:00 2025-10-27 11:25:01.721 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:25:01.729 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:01.734 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:25:01.794 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:01.824 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:25:01.827 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:25:01.859 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:01.943 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:25:01.976 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:01.979 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:01.714557+08:00 2025-10-27 11:25:02.070 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:02.110 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:25:02.110 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:25:02.250 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:25:02.261 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:25:02.278 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:02.281 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,91bc1aeb95c2d554] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9164, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:25:02.346 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:25:02.371 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:25:02.371 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:25:02.380 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.407 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.424 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.451 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.502 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.546 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.578 WARN 2458 --- [ocp-async-3,2c8d80aaa6f30dfd,c3a8df6139a313bc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:02.578 INFO 2458 --- [ocp-async-6,32430e699cd61ca7,68952d637d7df89c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:02.631 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.660 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:25:02.668 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:25:02.669 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:25:02.688 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,5014ea1dc69a708e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9163, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:25:02.707 INFO 2458 --- [prometheus-minute-schedule-1,746b47b72e2b7902,6f59547518ebf297] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:02.722 INFO 2458 --- [prometheus-minute-schedule-1,746b47b72e2b7902,6f59547518ebf297] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:02.809 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9105, status:SUCCESS 2025-10-27 11:25:02.825 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9105, status:SUCCESS 2025-10-27 11:25:02.834 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9106, status:SUCCESS 2025-10-27 11:25:02.836 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9106, status:SUCCESS 2025-10-27 11:25:03.374 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:03.375 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:03.894 INFO 2458 --- [ocp-schedules-27,34fed3f3c1e83a16,7a4d9216dada0ee8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:25:03.902 INFO 2458 --- [ocp-schedules-27,34fed3f3c1e83a16,7a4d9216dada0ee8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:03.946 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:03.947 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:25:03.967 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:03.967 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:25:03.982 INFO 2458 --- [ocp-schedules-27,34fed3f3c1e83a16,7a4d9216dada0ee8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:25:04.143 INFO 2458 --- [ocp-schedules-5,12a0be0326354fcf,f427820484b45c1e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:04.158 WARN 2458 --- [scanner-112184244-1,12a0be0326354fcf,304c0194c2b1ff2a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:04.284 INFO 2458 --- [alarm-detect-6,12a0be0326354fcf,5bbbc28adf88558d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535470 ~ 1761535494 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535485, value=56.711)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535485, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535485=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:25:04.579 INFO 2458 --- [ocp-async-10,893e61588d266318,790615a5ad41f0db] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:04.579 WARN 2458 --- [ocp-async-8,b14be89828438162,64e818f46efd7c04] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:04.974 INFO 2458 --- [ocp-schedules-26,df475fd4f593f4c6,4d3e4355d2e6f397] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:04.978 INFO 2458 --- [ocp-schedules-26,df475fd4f593f4c6,4d3e4355d2e6f397] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:05.707 INFO 2458 --- [prometheus-minute-schedule-1,02d4a2fd23a5cad7,f593d04856d56173] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:05.717 INFO 2458 --- [prometheus-minute-schedule-1,02d4a2fd23a5cad7,f593d04856d56173] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:05.946 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9b28750cb84709f5,e7e091981c0e6b7f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=9b28750cb84709f5, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:25:05.946 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,9b28750cb84709f5,e7e091981c0e6b7f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=9b28750cb84709f5, duration=1 ms] 2025-10-27 11:25:06.375 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:06.377 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:06.579 WARN 2458 --- [ocp-async-10,d5b797f57bf33728,5f8385ee35c89e09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:06.579 INFO 2458 --- [ocp-async-4,b918116aee8f923d,59a36d11a6cc1d33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:06.688 INFO 2458 --- [ocp-schedules-6,bfb043fb6227f0fa,21cdf73c67b2efe7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:07.359 INFO 2458 --- [ocp-schedules-4,00d4cfa457ad5e59,b06ef549c33b981e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:25:07.368 INFO 2458 --- [ocp-schedules-4,00d4cfa457ad5e59,b06ef549c33b981e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:07.368 INFO 2458 --- [ocp-schedules-4,00d4cfa457ad5e59,b06ef549c33b981e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:08.532 INFO 2458 --- [ocp-async-4,33f95509bc4cc17f,0bb9223c98a29941] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:08.579 WARN 2458 --- [ocp-async-7,5d8e31f551e70b5e,f2978284f6557341] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:08.579 INFO 2458 --- [ocp-async-7,ffc3958f56445428,6ba11ca6e7804235] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:08.707 INFO 2458 --- [prometheus-minute-schedule-1,c2c850a9d1083333,cc64653a0162cfd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:08.718 INFO 2458 --- [prometheus-minute-schedule-1,c2c850a9d1083333,cc64653a0162cfd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:08.788 INFO 2458 --- [ocp-schedules-2,ebb2d7c57a5ae3ea,2780a9b100a5208f] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:25:08.796 INFO 2458 --- [ocp-schedules-2,ebb2d7c57a5ae3ea,2780a9b100a5208f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:08.881 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:25:08.881 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.884 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:25:08.892 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.893 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:08.919 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.920 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:08.932 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:25:08.932 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.935 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:25:08.944 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.944 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:08.946 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.947 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:08.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:08.974 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.974 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:08.981 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:08.982 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:09.002 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.003 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:09.019 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.020 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:25:09.023 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.023 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:09.035 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.036 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.037 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.038 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.038 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.039 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.040 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.040 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.041 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.042 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.043 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.043 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.044 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:25:09.046 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:25:09.052 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.052 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:09.079 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.080 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:25:09.081 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.082 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:25:09.099 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:09.099 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.100 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:25:09.110 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.110 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:25:09.110 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.110 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:25:09.112 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:25:09.113 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:25:09.117 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.118 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:25:09.145 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.146 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:25:09.148 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.149 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:25:09.176 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.176 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:25:09.202 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.203 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:25:09.249 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.249 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.253 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.254 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.255 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:25:09.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.264 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:25:09.265 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:25:09.266 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:25:09.269 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:25:09.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:09.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:09.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:09.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:09.347 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:09.348 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:09.348 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:25:09.375 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:25:09.377 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:09.378 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:09.379 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:25:09.379 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:25:09.379 INFO 2458 --- [ocp-schedules-2,ebb2d7c57a5ae3ea,2780a9b100a5208f] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:25:10.015 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:25:10.015 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:25:10.060 INFO 2458 --- [Collect obproxy1,1fb884a51dc1328b,d7a72a008c2cad79] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:25:10.060 INFO 2458 --- [Collect obproxy1,1fb884a51dc1328b,d7a72a008c2cad79] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:25:10.060 INFO 2458 --- [Check obproxy1,c69d70c0367ac1cd,f1dcdfbc60121907] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:25:10.060 INFO 2458 --- [Check obproxy1,c69d70c0367ac1cd,f1dcdfbc60121907] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:25:10.099 INFO 2458 --- [Collect obproxy1,1fb884a51dc1328b,d7a72a008c2cad79] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:25:10.099 INFO 2458 --- [Check obproxy1,c69d70c0367ac1cd,f1dcdfbc60121907] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:25:10.103 INFO 2458 --- [Collect obproxy1,1fb884a51dc1328b,d7a72a008c2cad79] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:25:10.103 INFO 2458 --- [Check obproxy1,c69d70c0367ac1cd,f1dcdfbc60121907] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:25:10.103 INFO 2458 --- [Check obproxy1,c69d70c0367ac1cd,f1dcdfbc60121907] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:25:10.106 INFO 2458 --- [Collect obproxy1,1fb884a51dc1328b,d7a72a008c2cad79] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:25:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:25:10.579 WARN 2458 --- [ocp-async-9,673250dd7d59e4ab,6bb4569512c9a044] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:10.579 INFO 2458 --- [ocp-async-7,a15f35d565e7c81e,01c96a82fa672a44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:11.419 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535440, nodeCount=496, affectRows=496 2025-10-27 11:25:11.458 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535440, nodeCount=206, affectRows=206 2025-10-27 11:25:11.487 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535440, nodeCount=89, affectRows=89 2025-10-27 11:25:11.502 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535440, nodeCount=30, affectRows=30 2025-10-27 11:25:11.707 INFO 2458 --- [prometheus-minute-schedule-1,cb4672dac6533bee,61f716418bc9e829] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:11.715813+08:00, lastProcessTime=2025-10-27T11:25:01.714557+08:00 2025-10-27 11:25:11.718 INFO 2458 --- [prometheus-minute-schedule-1,cb4672dac6533bee,61f716418bc9e829] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:12.080 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:11.715813+08:00 2025-10-27 11:25:12.161 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=1000, affectRows=1000 2025-10-27 11:25:12.166 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=994, validCount=992 2025-10-27 11:25:12.222 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=1000, affectRows=1000 2025-10-27 11:25:12.226 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=999, validCount=995 2025-10-27 11:25:12.242 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505658) Duplicate entry '835370602979598336-1761535380' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505658) Duplicate entry '835370602979598336-1761535380' for key 'PRIMARY' 2025-10-27 11:25:12.378 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:12.379 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:12.485 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=1000, affectRows=1000 2025-10-27 11:25:12.489 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=992, validCount=990 2025-10-27 11:25:12.579 INFO 2458 --- [ocp-async-7,0656ce37a390a041,8dbe0d4dc7d4026b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:12.579 WARN 2458 --- [ocp-async-2,80fa997e4300592b,4d9e640184cb1ad0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:12.602 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=1000, affectRows=1000 2025-10-27 11:25:12.607 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=1000, validCount=994 2025-10-27 11:25:12.641 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=96, affectRows=96 2025-10-27 11:25:12.641 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=94, validCount=94 2025-10-27 11:25:14.290 INFO 2458 --- [ocp-schedules-1,66248f4a64b0554f,4c60aac616c21d5f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:14.319 WARN 2458 --- [scanner-112185958-1,66248f4a64b0554f,5dcbc5b5d058ee50] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:14.491 INFO 2458 --- [alarm-detect-19,66248f4a64b0554f,88959259bfcfa745] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535485 ~ 1761535504 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535500, value=60.844)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535500, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535500=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:25:14.579 WARN 2458 --- [ocp-async-1,cb424db03198eb05,f5607a8f729d1576] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:14.579 INFO 2458 --- [ocp-async-4,b40c9601f0553d75,bdcd1a0b63ff3b8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:14.707 INFO 2458 --- [prometheus-minute-schedule-1,376d74eeb9f7270a,fbbf75ea12c7b959] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:14.715 INFO 2458 --- [prometheus-minute-schedule-1,376d74eeb9f7270a,fbbf75ea12c7b959] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:14.985 INFO 2458 --- [ocp-schedules-17,8166501ecb3279a8,aafe20d5cfe0c715] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:14.989 INFO 2458 --- [ocp-schedules-17,8166501ecb3279a8,aafe20d5cfe0c715] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:15.379 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:15.380 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:15.553 INFO 2458 --- [ocp-schedules-27,bcfff063c8fc7f56,2489c2fb0befbd02] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:25:15.553+08:00, clusters=[1, 2] 2025-10-27 11:25:15.556 INFO 2458 --- [ocp-schedules-27,bcfff063c8fc7f56,2489c2fb0befbd02] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:25:15.558 INFO 2458 --- [ocp-schedules-27,bcfff063c8fc7f56,2489c2fb0befbd02] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:25:16.436 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535440, nodeCount=1000, affectRows=1000 2025-10-27 11:25:16.440 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535440, nodeCount=1000, validCount=988 2025-10-27 11:25:16.579 WARN 2458 --- [ocp-async-9,02fd9bf33726002c,95049947c0310b24] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:16.579 INFO 2458 --- [ocp-async-8,620a2fc71f479a89,08c35b8ff556578a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:16.698 INFO 2458 --- [ocp-schedules-29,840e91998c029ecb,d86bc1403701a0f1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:16.949 INFO 2458 --- [ocp-schedules-7,a398664fdc033614,4ded0f5cb7d6a6d3] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:25:16.949 INFO 2458 --- [ocp-schedules-7,a398664fdc033614,4ded0f5cb7d6a6d3] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:25:17.002 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:25:17.002 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:25:17.002 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:25:17.002 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:25:17.003 INFO 2458 --- [Collect all tenants major compactions6,a398664fdc033614,094cb6e25ac2487b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:25:17.003 INFO 2458 --- [Collect all tenants major compactions5,a398664fdc033614,9d1e8fa9d1fb60ed] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:25:17.037 INFO 2458 --- [Collect all tenants major compactions5,a398664fdc033614,9d1e8fa9d1fb60ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.038 INFO 2458 --- [Collect all tenants major compactions5,a398664fdc033614,9d1e8fa9d1fb60ed] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:25:17.038 INFO 2458 --- [Collect all tenants major compactions6,a398664fdc033614,094cb6e25ac2487b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.040 INFO 2458 --- [Collect all tenants major compactions6,a398664fdc033614,094cb6e25ac2487b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:25:17.042 INFO 2458 --- [Collect all tenants major compactions5,a398664fdc033614,9d1e8fa9d1fb60ed] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:25:17.044 INFO 2458 --- [Collect all tenants major compactions6,a398664fdc033614,094cb6e25ac2487b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:25:17.054 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.054 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.054 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.054 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.055 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.056 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:25:17.059 INFO 2458 --- [Collect all tenants major compactions3,a398664fdc033614,932d1124f0a51824] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:25:17.063 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.063 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.064 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.065 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:17.066 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:17.070 INFO 2458 --- [Collect all tenants major compactions1,a398664fdc033614,c2dbd4ee89202342] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:25:17.070 INFO 2458 --- [Collect all tenants major compactions2,a398664fdc033614,b8f2de231b9d964d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:25:17.070 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:25:17.073 INFO 2458 --- [Collect all tenants major compactions4,a398664fdc033614,683c232677afd8ac] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:25:17.074 INFO 2458 --- [ocp-schedules-7,a398664fdc033614,4ded0f5cb7d6a6d3] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:25:17.707 INFO 2458 --- [prometheus-minute-schedule-1,c59f1e0bcadeb488,b6f2ef9d16cee798] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:17.715 INFO 2458 --- [prometheus-minute-schedule-1,c59f1e0bcadeb488,b6f2ef9d16cee798] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:18.380 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:18.381 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:18.532 INFO 2458 --- [ocp-async-10,0f885170cc4bb6e4,d3b021479c41ad68] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:18.579 WARN 2458 --- [ocp-async-3,0811029d471c7845,1b4435b4ed738305] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:18.579 INFO 2458 --- [ocp-async-2,45e7d242c69df372,d4f38e1eb4afffd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:20.199 INFO 2458 --- [ocp-schedules-11,ddab55b56c180637,2b86be45402d29db] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:25:20.580 INFO 2458 --- [ocp-async-1,c8a9cece7f4e2e31,8ea29c97dd315150] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:20.580 WARN 2458 --- [ocp-async-6,349700a4f6744c2f,a5dae03d16f457e5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:20.707 INFO 2458 --- [prometheus-minute-schedule-1,f5c57ab542992fc9,bef33c18d81e88ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:20.714 INFO 2458 --- [prometheus-minute-schedule-1,f5c57ab542992fc9,bef33c18d81e88ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:20.750 INFO 2458 --- [ocp-schedules-5,c2afa75fcbb7b453,ca5c5fac6d53e7eb] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:25:20.750 INFO 2458 --- [ocp-schedules-5,c2afa75fcbb7b453,ca5c5fac6d53e7eb] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:25:20.757 INFO 2458 --- [ocp-schedules-5,c2afa75fcbb7b453,ca5c5fac6d53e7eb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:25:20.757 INFO 2458 --- [ocp-schedules-5,c2afa75fcbb7b453,ca5c5fac6d53e7eb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:25:21.381 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:21.382 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:21.410 INFO 2458 --- [ocp-schedules-26,8dc0dc3b085d4bfa,c9f26a8df19b3852] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:21.411 INFO 2458 --- [ocp-schedules-2,0595d1a8d58ebdce,ed08eb66c159eb14] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:25:21.413 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:25:21.411 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:25:21.413 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:25:21.413 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:25:21.413 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:25:21.426 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:21.426 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:21.426 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:21.426 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:21.428 INFO 2458 --- [ocp-schedules-22,c41cacc39eb7225a,1c084727549205c5] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:21.429 INFO 2458 --- [host-check-clockdiff2,c41cacc39eb7225a,1697fb47156b4641] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:25:21.430 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:21.430 INFO 2458 --- [host-check-clockdiff1,c41cacc39eb7225a,91a2d238cb3379fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:25:21.430 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:21.430 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:21.431 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:21.431 INFO 2458 --- [host-check-clockdiff3,c41cacc39eb7225a,bce80130bc5e088c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:25:21.431 INFO 2458 --- [host-check-clockdiff4,c41cacc39eb7225a,5c0d17f3cdf26b80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:25:21.431 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:21.432 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:21.433 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:21.433 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:21.433 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:21.451 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:21.451 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:21.451 INFO 2458 --- [host-check-clockdiff5,c41cacc39eb7225a,fb4ea8c75263fda4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:25:21.451 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:21.451 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:21.454 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:21.454 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:21.455 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:21.462 INFO 2458 --- [ocp-schedules-27,1feddfd56c15019a,b0844e90cc4df41c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:25:21.463 INFO 2458 --- [host-check-clockdiff1,c41cacc39eb7225a,91a2d238cb3379fd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535521 0 0 , error= 2025-10-27 11:25:21.468 INFO 2458 --- [host-check-clockdiff4,c41cacc39eb7225a,5c0d17f3cdf26b80] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535521 -1 -1 , error= 2025-10-27 11:25:21.475 INFO 2458 --- [host-check-clockdiff2,c41cacc39eb7225a,1697fb47156b4641] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535521 -1 -1 , error= 2025-10-27 11:25:21.478 INFO 2458 --- [host-check-clockdiff5,c41cacc39eb7225a,fb4ea8c75263fda4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535521 0 0 , error= 2025-10-27 11:25:21.511 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:25:21.512 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:25:21.513 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:25:21.511948580+08:00 2025-10-27 11:25:21.514 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:25:21.513+08:00 2025-10-27 11:25:21.515 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:25:21.515 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:25:21.515 INFO 2458 --- [host-check-agent-status4,0595d1a8d58ebdce,817ce5261c023bbe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:25:21.517 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:25:21.517 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:25:21.520 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:25:21.520 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:25:21.520 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:25:21.520 INFO 2458 --- [host-check-agent-status5,0595d1a8d58ebdce,eaf5fede74701efe] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:25:21.522 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:25:21.521924597+08:00 2025-10-27 11:25:21.522 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:25:21.522 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:25:21.522+08:00 2025-10-27 11:25:21.523 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:25:21.523 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:25:21.513+08:00 and status AVAILABLE 2025-10-27 11:25:21.525 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:25:21.525 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:25:21.525 INFO 2458 --- [host-check-agent-status2,0595d1a8d58ebdce,4de9f80cf3e63c8a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:25:21.526 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:25:21.526 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:25:21.526 INFO 2458 --- [host-check-agent-status3,0595d1a8d58ebdce,deb6c9c7049c2f44] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:25:21.528 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:25:21.528 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:25:21.529 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:25:21.528340337+08:00 2025-10-27 11:25:21.529 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:25:21.529+08:00 2025-10-27 11:25:21.529 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:25:21.522+08:00 and status AVAILABLE 2025-10-27 11:25:21.532 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:25:21.532 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:25:21.532 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:25:21.532 INFO 2458 --- [host-check-agent-status1,0595d1a8d58ebdce,3b895f153db1a200] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:25:21.532 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:25:21.532112167+08:00 2025-10-27 11:25:21.533 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:25:21.532+08:00 2025-10-27 11:25:21.534 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:25:21.538 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:25:21.529+08:00 and status AVAILABLE 2025-10-27 11:25:21.539 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:25:21.540 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:25:21.540285749+08:00 2025-10-27 11:25:21.541 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:25:21.540+08:00 2025-10-27 11:25:21.543 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:25:21.544 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:25:21.532+08:00 and status AVAILABLE 2025-10-27 11:25:21.545 INFO 2458 --- [ocp-schedules-10,da5d7cc630984ab8,78a754c19f539e91] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:25:21.548 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:25:21.540+08:00 and status AVAILABLE 2025-10-27 11:25:21.548 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:25:21.549 INFO 2458 --- [ocp-schedules-11,d891995b4a7486bc,2a4fda594a84d632] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:25:21.550 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:21.552 INFO 2458 --- [ocp-schedules-10,da5d7cc630984ab8,78a754c19f539e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:21.552 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:25:21.552 INFO 2458 --- [ocp-schedules-11,d891995b4a7486bc,2a4fda594a84d632] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:21.553 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:21.553 INFO 2458 --- [ocp-schedules-11,d891995b4a7486bc,2a4fda594a84d632] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:25:21.553 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:21.554 INFO 2458 --- [ocp-schedules-15,d66c48367ab1dda7,b964786af9d346fe] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:25:21.556 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:21.558 INFO 2458 --- [ocp-schedules-15,d66c48367ab1dda7,b964786af9d346fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:21.566 INFO 2458 --- [ocp-schedules-14,790fc6ca52f74fea,9b4709c37c2a9f4c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:25:21.567 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:21.571 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:21.574 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:21.574 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:21.577 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:21.578 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:21.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:25:21.611 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:21.618 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.621 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:21.621 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.622 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:21.626 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:25:21.626 INFO 2458 --- [ocp-schedules-10,da5d7cc630984ab8,78a754c19f539e91] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:25:21.631 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:21.631 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:25:21.633 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:21.635 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:21.647 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:25:21.650 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:25:21.651 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:25:21.651 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:25:21.675 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:25:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:21.683 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:25:21.690 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:25:21.690 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:25:21.695 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:25:21.695 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:25:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:21.714146+08:00, lastProcessTime=2025-10-27T11:25:11.715813+08:00 2025-10-27 11:25:21.719 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:21.723 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:21.725 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:21.725 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:21.727 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:21.739 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:25:21.740 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:25:21.741 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:25:21.749 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:25:21.760 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:21.760 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.760 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.760 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.760 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.761 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.761 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:25:21.761 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:21.762 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:25:21.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:21.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:25:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:21.776 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:25:21.781 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:25:21.782 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:25:21.788 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:25:21.791 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:21.791 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:21.794 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:25:21.806 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:21.813 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:21.813 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:21.818 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:21.825 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:25:21.825 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:21.827 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:25:21.833 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:25:21.838 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:25:21.856 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:25:21.857 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:21.858 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.859 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:21.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.863 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.864 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:21.865 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.868 INFO 2458 --- [host-check-host-status4,8dc0dc3b085d4bfa,da4a65941f04728d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:25:21.869 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:25:21.870 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:21.870 INFO 2458 --- [host-check-host-status2,8dc0dc3b085d4bfa,24761426219bd45d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:25:21.871 INFO 2458 --- [host-check-host-status5,8dc0dc3b085d4bfa,fa1403f7adbfa082] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:25:21.875 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:25:21.879 INFO 2458 --- [host-check-host-status3,8dc0dc3b085d4bfa,0f01a8193714e487] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:25:21.884 INFO 2458 --- [host-check-host-status1,8dc0dc3b085d4bfa,d025eb6166b057d8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:25:21.885 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:21.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:21.937 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:25:21.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:21.962 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:21.969 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:21.714146+08:00 2025-10-27 11:25:21.985 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:25:21.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:25:21.988 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:21.989 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:25:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:22.007 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:25:22.031 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:22.031 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:22.032 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:25:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:22.037 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:22.058 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:25:22.058 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:25:22.071 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:22.071 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:22.116 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:22.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:25:22.168 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:25:22.179 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:22.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:22.228 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:25:22.275 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:25:22.275 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.275 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.276 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:22.276 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:25:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.278 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:25:22.313 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:22.352 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:25:22.390 INFO 2458 --- [host-check-clockdiff3,c41cacc39eb7225a,bce80130bc5e088c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535522 0 0 , error= 2025-10-27 11:25:22.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:25:22.406 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:25:22.406 INFO 2458 --- [ocp-schedules-15,d66c48367ab1dda7,b964786af9d346fe] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:25:22.580 WARN 2458 --- [ocp-async-9,5afc45728a5da481,db61bbc85d041ffe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:22.580 INFO 2458 --- [ocp-async-8,96e57f613e7f14cc,8010a8f0acfb6bda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:22.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,711f3eac699951eb,da64648960b7beda] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=711f3eac699951eb, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:25:22.817 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,711f3eac699951eb,da64648960b7beda] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=711f3eac699951eb, duration=1 ms] 2025-10-27 11:25:23.707 INFO 2458 --- [prometheus-minute-schedule-1,bddb525ae8a636b8,5bfae943e506aef7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:23.717 INFO 2458 --- [prometheus-minute-schedule-1,bddb525ae8a636b8,5bfae943e506aef7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:24.383 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:24.383 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:24.499 INFO 2458 --- [ocp-schedules-28,dc8143dcdbb97e44,167e5810c082b72a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:24.534 WARN 2458 --- [scanner-112186240-1,dc8143dcdbb97e44,4cd911904a4226b2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:24.601 WARN 2458 --- [ocp-async-10,f06f58b426efcf68,6973949838f77f62] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:24.601 INFO 2458 --- [ocp-async-3,5bb7b9d3cd36db01,693da6ab7be99553] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:24.999 INFO 2458 --- [ocp-schedules-3,394d2f584275660f,0fda4fef293f91e4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:25.002 INFO 2458 --- [ocp-schedules-3,394d2f584275660f,0fda4fef293f91e4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:26.601 WARN 2458 --- [ocp-async-2,2f8a8ba3471d46c2,0b29dd8c41b7f2b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:26.601 INFO 2458 --- [ocp-async-6,9c64acfb0193d09e,6588518ab8b9b7cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:26.705 INFO 2458 --- [ocp-schedules-11,ed22721774a804b8,736d037e024a0cd4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:26.707 INFO 2458 --- [prometheus-minute-schedule-1,a7ada48174fa9fa3,b1feb58973acdaa8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:26.713 INFO 2458 --- [prometheus-minute-schedule-1,a7ada48174fa9fa3,b1feb58973acdaa8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:27.371 INFO 2458 --- [ocp-schedules-13,fa4f7b6e1d50d53b,bf5919d7dfc0485e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:25:27.384 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:27.384 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:27.408 INFO 2458 --- [ocp-schedules-13,fa4f7b6e1d50d53b,bf5919d7dfc0485e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:27.408 INFO 2458 --- [ocp-schedules-13,fa4f7b6e1d50d53b,bf5919d7dfc0485e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:28.532 INFO 2458 --- [ocp-async-6,34603b68b77104d2,21f57a9331df20e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:28.601 WARN 2458 --- [ocp-async-9,aef0168426fdaedd,926cd505b72af79d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:28.601 INFO 2458 --- [ocp-async-5,8349d4e20357b57a,b87010adb723d3f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:28.991 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:25:28.991 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:25:28.993 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:25:28.993 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:25:28.995 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:25:28.995 INFO 2458 --- [ocp-schedules-29,7cad9007bed8e542,17769599f7e3859e] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:25:29.450 INFO 2458 --- [ocp-schedules-9,290c1a40a1bb0b7c,16bacb4c02f079c3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:25:29.454 INFO 2458 --- [ocp-schedules-9,290c1a40a1bb0b7c,16bacb4c02f079c3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:25:29.454 INFO 2458 --- [ocp-schedules-9,290c1a40a1bb0b7c,16bacb4c02f079c3] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:25:29.707 INFO 2458 --- [prometheus-minute-schedule-1,8770e55a81926d84,d937b75a62159e14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:29.715 INFO 2458 --- [prometheus-minute-schedule-1,8770e55a81926d84,d937b75a62159e14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:30.384 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:30.385 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:30.602 WARN 2458 --- [ocp-async-7,d1465b98765a7781,c37465d8c8954fc6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:30.602 INFO 2458 --- [ocp-async-7,6f2538ee8c0de10d,5cd995e9bb602c32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:31.713862+08:00, lastProcessTime=2025-10-27T11:25:21.714146+08:00 2025-10-27 11:25:31.933 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:31.713862+08:00 2025-10-27 11:25:32.602 INFO 2458 --- [ocp-async-2,f6fb6038955cfb23,68b1c06b39aa1096] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:32.602 WARN 2458 --- [ocp-async-1,bea82d9371d7dcae,638372eef04dc69b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:32.707 INFO 2458 --- [prometheus-minute-schedule-1,393079c5a49f4882,9b4eaf233fed147c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:32.715 INFO 2458 --- [prometheus-minute-schedule-1,393079c5a49f4882,9b4eaf233fed147c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:33.386 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:33.386 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:33.904 INFO 2458 --- [ocp-schedules-17,15fd2c96596374b8,a6b4b481c1e5d1c0] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:25:33.904 INFO 2458 --- [ocp-schedules-17,15fd2c96596374b8,a6b4b481c1e5d1c0] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:25:34.602 INFO 2458 --- [ocp-async-6,b4ce43b60a8ccea1,5f22c5c5add7dcbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:34.602 WARN 2458 --- [ocp-async-8,faf6b1208f479dbb,df4769676a72124f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:34.656 INFO 2458 --- [ocp-schedules-28,4d92835fde806d62,a087e760aae8973d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:34.668 WARN 2458 --- [scanner-112186396-1,4d92835fde806d62,62c7b1169517daaa] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:34.848 INFO 2458 --- [alarm-detect-12,4d92835fde806d62,e1c01b11458cf7be] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535500 ~ 1761535524 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535515, value=45.966)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535515, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535515=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:25:35.009 INFO 2458 --- [ocp-schedules-7,f0700d1f22d56eb5,7f7f6557814a34cc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:35.012 INFO 2458 --- [ocp-schedules-7,f0700d1f22d56eb5,7f7f6557814a34cc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:35.707 INFO 2458 --- [prometheus-minute-schedule-1,e68076f19e67ab67,cbb705dbe0bbec72] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:35.714 INFO 2458 --- [prometheus-minute-schedule-1,e68076f19e67ab67,cbb705dbe0bbec72] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:36.387 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:36.387 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:36.602 INFO 2458 --- [ocp-async-5,e4da2be5c93de065,ac08d5532a35ed2e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:36.602 WARN 2458 --- [ocp-async-3,2915a2d686d32911,58c7d7ade0daca1d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:36.712 INFO 2458 --- [ocp-schedules-19,cf5bbc5e0f5f132c,4b1e21cb8ec65144] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:37.708 INFO 2458 --- [prometheus-cache-monitor1,d9b68d5c85679895,93dcb332448f34f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20386,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:25:37.708 INFO 2458 --- [prometheus-cache-monitor1,d9b68d5c85679895,93dcb332448f34f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:25:38.532 INFO 2458 --- [ocp-async-5,59f44a341bfda9f7,b5c0b6d733736f81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:38.558 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:25:38.558 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:25:38.602 INFO 2458 --- [ocp-async-1,07dbc8dd7b46d212,c980bf774889ebda] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:38.602 WARN 2458 --- [ocp-async-4,4675dde11ddc2e54,95bce0a58b01f4fa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:38.619 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:38.619 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:38.661 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:38.661 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:38.661 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:38.662 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:25:38.675 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:38.676 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:38.676 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:38.677 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:25:38.707 INFO 2458 --- [prometheus-minute-schedule-1,df2337d3ed39f0a3,d6833206be57b6d2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:38.714 INFO 2458 --- [prometheus-minute-schedule-1,df2337d3ed39f0a3,d6833206be57b6d2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:38.725 INFO 2458 --- [ocp-schedules-12,5c8cc527befbbbca,488cd8d6413a851f] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:25:38.740 INFO 2458 --- [ocp-schedules-12,5c8cc527befbbbca,488cd8d6413a851f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:38.741 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:25:38.741 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:25:38.741 INFO 2458 --- [ocp-schedules-12,5c8cc527befbbbca,488cd8d6413a851f] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:25:38.759 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:38.760 INFO 2458 --- [ocp-schedules-1,99b903ac67d34f6b,d943635d41a0da38] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:25:39.100 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:39.100 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:25:39.100 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:25:39.100 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:25:39.100 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:25:39.100 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:25:39.112 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:39.113 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:39.113 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:39.114 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:39.115 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:39.116 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:39.117 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:39.117 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:39.118 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:39.118 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:39.169 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:25:39.169 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:25:39.171 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:25:39.173 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:25:39.177 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:25:39.182 INFO 2458 --- [ocp-schedules-26,eaff668dbbc886a9,7e88df93ea8656b6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:25:39.388 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:39.388 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:40.012 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:25:40.012 INFO 2458 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:25:40.048 INFO 2458 --- [Check obproxy1,3522cfdd06b8cd54,32ded36182287c7f] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:25:40.048 INFO 2458 --- [Check obproxy1,3522cfdd06b8cd54,32ded36182287c7f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:25:40.049 INFO 2458 --- [Collect obproxy1,ce677a5a9705f113,49fcb18eaefd4c91] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:25:40.049 INFO 2458 --- [Collect obproxy1,ce677a5a9705f113,49fcb18eaefd4c91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:25:40.084 INFO 2458 --- [Collect obproxy1,ce677a5a9705f113,49fcb18eaefd4c91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:25:40.084 INFO 2458 --- [Check obproxy1,3522cfdd06b8cd54,32ded36182287c7f] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:25:40.088 INFO 2458 --- [Collect obproxy1,ce677a5a9705f113,49fcb18eaefd4c91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:25:40.088 INFO 2458 --- [Check obproxy1,3522cfdd06b8cd54,32ded36182287c7f] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:25:40.088 INFO 2458 --- [Check obproxy1,3522cfdd06b8cd54,32ded36182287c7f] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:25:40.090 INFO 2458 --- [Collect obproxy1,ce677a5a9705f113,49fcb18eaefd4c91] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:25:40.602 WARN 2458 --- [ocp-async-8,1f11dd2e95d9fbbb,537fb89371150ac4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:40.602 INFO 2458 --- [ocp-async-8,2764f419500fd004,131e2f20c0c92001] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:41.190 INFO 2458 --- [ocp-schedules-25,fa4af468d3ba3029,957cfc7ba6e3531a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:25:41.190 INFO 2458 --- [ocp-schedules-25,fa4af468d3ba3029,957cfc7ba6e3531a] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:25:41.707 INFO 2458 --- [prometheus-minute-schedule-1,a591c1073eb145d1,a03f2235e83717f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:41.714 INFO 2458 --- [prometheus-minute-schedule-1,a591c1073eb145d1,a03f2235e83717f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:41.714036+08:00, lastProcessTime=2025-10-27T11:25:31.713862+08:00 2025-10-27 11:25:41.920 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:41.714036+08:00 2025-10-27 11:25:42.389 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:42.389 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:42.602 INFO 2458 --- [ocp-async-10,15ecddf69b3894a7,5804b621bb625175] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:42.602 WARN 2458 --- [ocp-async-3,ec8d75b5a6f926b4,3138cdc5b7cf85c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:44.200 INFO 2458 --- [ocp-schedules-25,f73f15de86de23ee,1c333d3521c258f9] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:25:44.603 INFO 2458 --- [ocp-async-4,ba5f69f69e143340,2f9f98b6cd06fa22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:44.603 WARN 2458 --- [ocp-async-7,563689f175a867ed,34db8e37f06549f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:44.707 INFO 2458 --- [prometheus-minute-schedule-1,f33b553807835f74,0f7a554cc24a7ee4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:44.714 INFO 2458 --- [prometheus-minute-schedule-1,f33b553807835f74,0f7a554cc24a7ee4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:44.855 INFO 2458 --- [ocp-schedules-15,01c90c677dc28d68,a76592c2556035fc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:44.888 WARN 2458 --- [scanner-112186666-1,01c90c677dc28d68,926e2bd850988cc9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:44.980 INFO 2458 --- [alarm-detect-14,01c90c677dc28d68,637d174bb8be8cb6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535515 ~ 1761535534 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535530, value=46.491)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535530, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535530=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:25:45.019 INFO 2458 --- [ocp-schedules-10,47471e9107b486af,472dcb1af60db20a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:45.027 INFO 2458 --- [ocp-schedules-10,47471e9107b486af,472dcb1af60db20a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:45.102 INFO 2458 --- [prometheus-second-schedule-1,0813a5eb40fe9b27,96e102225cc05727] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:25:45.389 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:45.390 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:46.603 INFO 2458 --- [ocp-async-1,f600e9bf1d70b7c6,8f3c6cfb6fb3f537] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:46.603 WARN 2458 --- [ocp-async-6,a6e1c1fa12fd53df,16099bdee517d5e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:46.721 INFO 2458 --- [ocp-schedules-12,822c9b6c1439588b,fbc263ad725138b9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:47.411 INFO 2458 --- [ocp-schedules-14,c88f3c0f78144afe,a326be4e0d29ea68] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:25:47.419 INFO 2458 --- [ocp-schedules-14,c88f3c0f78144afe,a326be4e0d29ea68] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:47.419 INFO 2458 --- [ocp-schedules-14,c88f3c0f78144afe,a326be4e0d29ea68] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:25:47.707 INFO 2458 --- [prometheus-minute-schedule-1,52472a17163f1011,0c3a559ca65a4321] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:47.714 INFO 2458 --- [prometheus-minute-schedule-1,52472a17163f1011,0c3a559ca65a4321] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:48.390 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:48.391 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:48.533 INFO 2458 --- [ocp-async-1,a841bbaa8a3b1411,8c1d9928574491e3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:48.603 WARN 2458 --- [ocp-async-5,9324b86ccaf3bc1f,7e2d91bc5e5deb00] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:48.603 INFO 2458 --- [ocp-async-3,79428e9f12fae3b5,135778dd44a13f90] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:49.270 INFO 2458 --- [ocp-schedules-30,ec60a7fde0cbd6d2,f7da7a1d807565ed] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:25:49.270+08:00, clusters=[1, 2] 2025-10-27 11:25:49.284 INFO 2458 --- [ocp-schedules-30,ec60a7fde0cbd6d2,f7da7a1d807565ed] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:25:49.288 INFO 2458 --- [ocp-schedules-30,ec60a7fde0cbd6d2,f7da7a1d807565ed] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:25:50.603 WARN 2458 --- [ocp-async-7,ab5832ac9c01282e,04d4b8847dddc234] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:50.603 INFO 2458 --- [ocp-async-2,9c622d9c8e28fc01,b5c3c5e7249ad884] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:50.707 INFO 2458 --- [prometheus-minute-schedule-1,3f297a3986dbad39,afef13975b4da087] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:50.715 INFO 2458 --- [prometheus-minute-schedule-1,3f297a3986dbad39,afef13975b4da087] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:50.902 INFO 2458 --- [ocp-schedules-13,54e4a09ed3f0e3d0,1fa6cd6fe206be4c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:25:50.902 INFO 2458 --- [ocp-schedules-13,54e4a09ed3f0e3d0,1fa6cd6fe206be4c] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:25:50.910 INFO 2458 --- [ocp-schedules-13,54e4a09ed3f0e3d0,1fa6cd6fe206be4c] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:25:50.910 INFO 2458 --- [ocp-schedules-13,54e4a09ed3f0e3d0,1fa6cd6fe206be4c] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:25:51.391 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:51.392 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:51.410 INFO 2458 --- [ocp-schedules-17,56bb929af2ff4048,76745bde92f45104] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:51.410 INFO 2458 --- [ocp-schedules-5,df9647b0b221ad78,486c8e6fc09b657e] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:25:51.411 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:25:51.412 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:25:51.412 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:25:51.412 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:25:51.426 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:51.426 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:51.426 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:51.427 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:51.427 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:51.427 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:51.428 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:51.431 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:51.431 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:51.432 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:51.433 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:51.433 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:51.434 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:51.436 INFO 2458 --- [ocp-schedules-9,77f798a134d76b56,4effe676cd1e1324] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:25:51.436 INFO 2458 --- [host-check-clockdiff1,77f798a134d76b56,46e08f47ca7b0dd2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:25:51.436 INFO 2458 --- [host-check-clockdiff2,77f798a134d76b56,d9ca96085f7ac05f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:25:51.436 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:51.436 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:51.437 INFO 2458 --- [host-check-clockdiff3,77f798a134d76b56,d84c9ff24e074426] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:25:51.437 INFO 2458 --- [host-check-clockdiff4,77f798a134d76b56,2ff822a6795a0b94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:25:51.438 INFO 2458 --- [host-check-clockdiff5,77f798a134d76b56,2052e1619ea1ce15] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:25:51.442 INFO 2458 --- [ocp-schedules-8,2a80b2e8a3e113ca,3d446c7810e0d8df] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:25:51.466 INFO 2458 --- [host-check-clockdiff1,77f798a134d76b56,46e08f47ca7b0dd2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535551 0 0 , error= 2025-10-27 11:25:51.466 INFO 2458 --- [host-check-clockdiff4,77f798a134d76b56,2ff822a6795a0b94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535551 0 0 , error= 2025-10-27 11:25:51.469 INFO 2458 --- [host-check-clockdiff3,77f798a134d76b56,d84c9ff24e074426] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535551 -1 -1 , error= 2025-10-27 11:25:51.469 INFO 2458 --- [host-check-clockdiff2,77f798a134d76b56,d9ca96085f7ac05f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535551 -1 -1 , error= 2025-10-27 11:25:51.474 INFO 2458 --- [host-check-clockdiff5,77f798a134d76b56,2052e1619ea1ce15] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535551 0 0 , error= 2025-10-27 11:25:51.510 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:25:51.511 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:25:51.511 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:25:51.512 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:25:51.512 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:25:51.514 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status4,56bb929af2ff4048,8e839c91c5e80619] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status3,56bb929af2ff4048,684712c8e0d538b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-agent-status2,56bb929af2ff4048,651c0471b80a8a0a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:25:51.515 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:25:51.515183817+08:00 2025-10-27 11:25:51.516 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:25:51.514389985+08:00 2025-10-27 11:25:51.516 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:25:51.515+08:00 2025-10-27 11:25:51.516 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:25:51.516+08:00 2025-10-27 11:25:51.516 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:25:51.515590914+08:00 2025-10-27 11:25:51.517 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:25:51.516+08:00 2025-10-27 11:25:51.517 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:25:51.517 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:25:51.518 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:25:51.517615183+08:00 2025-10-27 11:25:51.519 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:25:51.518+08:00 2025-10-27 11:25:51.519 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:25:51.519620874+08:00 2025-10-27 11:25:51.520 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:25:51.519+08:00 2025-10-27 11:25:51.523 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:25:51.523 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:25:51.515+08:00 and status AVAILABLE 2025-10-27 11:25:51.524 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:25:51.516+08:00 and status AVAILABLE 2025-10-27 11:25:51.525 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:25:51.516+08:00 and status AVAILABLE 2025-10-27 11:25:51.526 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:25:51.526 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:25:51.526 INFO 2458 --- [host-check-agent-status5,56bb929af2ff4048,5e50e1ca0ade7e24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:25:51.526 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:25:51.518+08:00 and status AVAILABLE 2025-10-27 11:25:51.527 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:25:51.519+08:00 and status AVAILABLE 2025-10-27 11:25:51.528 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:25:51.528 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:25:51.529 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:25:51.529 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:25:51.531 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:25:51.531 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:25:51.531 INFO 2458 --- [host-check-agent-status1,56bb929af2ff4048,93a0f07e7fa36a21] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:25:51.531 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:25:51.533 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:25:51.545 INFO 2458 --- [ocp-schedules-18,e20811be85169c3e,0a505a7192eb8958] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:25:51.548 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:25:51.548 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:51.549 INFO 2458 --- [ocp-schedules-10,23256af3ef677fb0,350f1ca8b9f59e4b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:25:51.549 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:25:51.551 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:25:51.551 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:51.551 INFO 2458 --- [ocp-schedules-18,e20811be85169c3e,0a505a7192eb8958] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:51.551 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:25:51.552 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:25:51.552 INFO 2458 --- [ocp-schedules-10,23256af3ef677fb0,350f1ca8b9f59e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:51.553 INFO 2458 --- [ocp-schedules-10,23256af3ef677fb0,350f1ca8b9f59e4b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:25:51.553 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:25:51.554 INFO 2458 --- [ocp-schedules-19,26f30c5e0c8bddb2,733aa9ee5cfffa8a] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:25:51.554 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:51.557 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:51.558 INFO 2458 --- [ocp-schedules-19,26f30c5e0c8bddb2,733aa9ee5cfffa8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:51.565 INFO 2458 --- [ocp-schedules-16,2b2521f900ed5b8a,b4e8fa51c83b4175] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:25:51.568 INFO 2458 --- [ocp-schedules-3,4717a3ec3bdbffc6,ec96e5770672d264] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:25:51.569 INFO 2458 --- [ocp-schedules-12,a2335315d38fc85a,50021ded9799c75e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:25:51.578 INFO 2458 --- [ocp-schedules-12,a2335315d38fc85a,50021ded9799c75e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:25:51.580 INFO 2458 --- [ocp-schedules-3,4717a3ec3bdbffc6,ec96e5770672d264] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:25:51.607 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.609 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:51.610 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:25:51.629 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:51.633 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:51.633 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:51.635 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:51.640 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:25:51.649 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:25:51.649 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:25:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:25:51.652 INFO 2458 --- [ocp-schedules-18,e20811be85169c3e,0a505a7192eb8958] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:25:51.653 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:25:51.655 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:25:51.665 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.666 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:51.673 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:25:51.693 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:25:51.695 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:25:51.695 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:25:51.702 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:25:51.709 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:25:51.709 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.709 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.710 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.711 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:25:51.714125+08:00, lastProcessTime=2025-10-27T11:25:41.714036+08:00 2025-10-27 11:25:51.724 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:51.725 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.726 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:25:51.730 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:51.731 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:51.735 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:51.737 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:51.742 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:25:51.748 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:25:51.750 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:25:51.751 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:25:51.758 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:25:51.775 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:25:51.790 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:25:51.792 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:25:51.792 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:25:51.795 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:25:51.797 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:51.797 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.798 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:51.799 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.802 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:25:51.803 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:51.809 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:25:51.813 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:51.813 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.813 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:25:51.821 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:51.824 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:51.830 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:51.832 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:51.836 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:25:51.842 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:25:51.844 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:25:51.848 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:25:51.851 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:25:51.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:51.869 INFO 2458 --- [host-check-host-status4,df9647b0b221ad78,1844b774d1fffffb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:25:51.873 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:25:51.876 INFO 2458 --- [host-check-host-status2,df9647b0b221ad78,052fdf5d5bd5d59e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:25:51.882 INFO 2458 --- [host-check-host-status5,df9647b0b221ad78,9f094a3e6795517c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:25:51.887 INFO 2458 --- [host-check-host-status3,df9647b0b221ad78,79fa0fdfd6bfe708] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:25:51.898 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:51.899 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.899 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:51.900 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:25:51.904 INFO 2458 --- [host-check-host-status1,df9647b0b221ad78,d43cb0cfd6d3f756] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:25:51.909 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:25:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:25:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:51.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:25:51.919 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:25:51.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:51.921 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:25:51.956 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:51.956 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:51.971 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:25:51.972 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:25:51.714125+08:00 2025-10-27 11:25:51.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:51.993 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:52.013 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:25:52.014 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.015 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:25:52.016 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.017 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:25:52.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:52.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:52.033 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:25:52.057 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:52.058 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.059 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:25:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:25:52.089 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:25:52.089 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:25:52.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:52.096 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:25:52.133 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:25:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:25:52.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:25:52.195 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:25:52.204 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:25:52.207 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:25:52.256 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:25:52.308 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:25:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.309 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:25:52.311 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.311 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.311 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.312 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:25:52.347 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:25:52.396 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:25:52.396 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.396 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:25:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:25:52.397 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:25:52.398 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:25:52.459 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:25:52.459 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:25:52.459 INFO 2458 --- [ocp-schedules-19,26f30c5e0c8bddb2,733aa9ee5cfffa8a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:25:52.603 WARN 2458 --- [ocp-async-8,164ea0d5721e5e9f,9fbfefe3a1ee0085] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:52.603 INFO 2458 --- [ocp-async-8,f2228aea38c27f17,bb8a9eeab3adfe33] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:25:53.234 INFO 2458 --- [ocp-schedules-12,dc7f3c1bde72bbf2,eec24fb6d1fe2ec0] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20386,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:25:53.235 INFO 2458 --- [ocp-schedules-12,dc7f3c1bde72bbf2,eec24fb6d1fe2ec0] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:25:53.707 INFO 2458 --- [prometheus-minute-schedule-1,5825963b7e3e4b84,905888a2eca9cc5a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:53.715 INFO 2458 --- [prometheus-minute-schedule-1,5825963b7e3e4b84,905888a2eca9cc5a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:54.393 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:54.394 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:54.603 WARN 2458 --- [ocp-async-3,cbe0deef72335a46,d16e2a66770cde32] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:54.603 INFO 2458 --- [ocp-async-4,95636ac940b8c931,a50273279b481d3a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:54.985 INFO 2458 --- [ocp-schedules-23,9543ef25452f92ce,ab802a892a69f39f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:25:55.034 INFO 2458 --- [ocp-schedules-1,dfab20af68cf3266,02624222bdf5a329] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:25:55.037 WARN 2458 --- [scanner-112186847-1,9543ef25452f92ce,48d3b728169c2633] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:25:55.039 INFO 2458 --- [ocp-schedules-1,dfab20af68cf3266,02624222bdf5a329] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:25:55.092 INFO 2458 --- [alarm-detect-15,9543ef25452f92ce,2c54a3d19cfb9938] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535485 ~ 1761535545 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535500, value=203834.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203834.56, values={1761535500=203834.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:25:55.145 INFO 2458 --- [alarm-detect-19,9543ef25452f92ce,7cbbff8953a1517c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535485 ~ 1761535545 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535500, value=93.141)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.141, values={1761535500=93.141}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:25:55.162 INFO 2458 --- [alarm-detect-8,9543ef25452f92ce,615b1b60cf545fba] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535485 ~ 1761535545 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535500, value=203835.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203835.912, values={1761535500=203835.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:25:55.178 INFO 2458 --- [alarm-detect-12,9543ef25452f92ce,03e93d9b988c3b5d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535530 ~ 1761535545 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535545, value=51.274)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535545, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535545=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:25:55.278 INFO 2458 --- [ocp-schedules-2,16948e1cd89792b4,d9410457a011efe9] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:25:56.603 WARN 2458 --- [ocp-async-6,f49b1058a0f63720,2ae2d7c67a1c9e11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:56.603 INFO 2458 --- [ocp-async-2,f6a71f5686c9bb3a,072c58ae7d646e20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:56.707 INFO 2458 --- [prometheus-minute-schedule-1,d5d6b6564ed8198c,acf787b8959e02ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:56.714 INFO 2458 --- [prometheus-minute-schedule-1,d5d6b6564ed8198c,acf787b8959e02ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:25:56.731 INFO 2458 --- [ocp-schedules-4,b516ba931dcff829,b74288b6cd949c06] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:25:57.394 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:25:57.394 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:25:58.190 INFO 2458 --- [ocp-schedules-25,27c6554f615d00b9,8c9e8616b847df69] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:25:58.198 INFO 2458 --- [ocp-schedules-25,27c6554f615d00b9,8c9e8616b847df69] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:25:58.240 INFO 2458 --- [ocp-schedules-25,27c6554f615d00b9,8c9e8616b847df69] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:25:58.533 INFO 2458 --- [ocp-async-2,29e5adcd33a9fbbe,bad17b2219f51ffc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:25:58.604 WARN 2458 --- [ocp-async-1,04dbe2f899383cad,64f4358ef0f39e9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:25:58.604 INFO 2458 --- [ocp-async-8,5986a1d9b7e0009a,ab753e3a1d3e9828] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:25:59.434 INFO 2458 --- [ocp-schedules-12,024116ba9e6aea3d,980e8973ea810ac7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:25:59.708 INFO 2458 --- [prometheus-minute-schedule-1,56430e003040aedf,139c8058f67cdc14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535560, interval=60 2025-10-27 11:25:59.716 INFO 2458 --- [prometheus-minute-schedule-1,56430e003040aedf,139c8058f67cdc14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:00.395 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:00.395 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:00.604 WARN 2458 --- [ocp-async-3,b6c6113b55604de8,cd82b2b56d0e1872] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:00.604 INFO 2458 --- [ocp-async-5,23b31bd3e1ab64dd,a9492fcb3223884b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:01.714994+08:00, lastProcessTime=2025-10-27T11:25:51.714125+08:00 2025-10-27 11:26:01.947 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:01.714994+08:00 2025-10-27 11:26:02.604 WARN 2458 --- [ocp-async-6,25e2c69da69c86a9,7a6432a63ded94cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:02.604 INFO 2458 --- [ocp-async-2,822027143fdabfc0,77d5107ed80a4c36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:02.707 INFO 2458 --- [prometheus-minute-schedule-1,ea62a3cb654ce48b,a63cf3db45b600d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:02.715 INFO 2458 --- [prometheus-minute-schedule-1,ea62a3cb654ce48b,a63cf3db45b600d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:03.395 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:03.397 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:03.982 INFO 2458 --- [ocp-schedules-13,5604a30d19435db2,09030aa22be3337c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:26:03.990 INFO 2458 --- [ocp-schedules-13,5604a30d19435db2,09030aa22be3337c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:04.028 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:04.029 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:26:04.044 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:04.044 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:04.045 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:04.045 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:04.045 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:04.045 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:26:04.059 INFO 2458 --- [ocp-schedules-13,5604a30d19435db2,09030aa22be3337c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:26:04.304 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,9d83c2defe6230e3,c42b02f4468d7d7d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=9d83c2defe6230e3, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:26:04.305 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,9d83c2defe6230e3,c42b02f4468d7d7d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=9d83c2defe6230e3, duration=1 ms] 2025-10-27 11:26:04.604 WARN 2458 --- [ocp-async-1,52ae0a4d2b8bb21a,ed29b31ba47849ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:04.604 INFO 2458 --- [ocp-async-1,6ce01f6b3fae84a9,369254dbbe03c742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:05.047 INFO 2458 --- [ocp-schedules-18,5a63ddfcfc73942a,b2387db148979b8e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:05.051 INFO 2458 --- [ocp-schedules-18,5a63ddfcfc73942a,b2387db148979b8e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:05.185 INFO 2458 --- [ocp-schedules-22,762910a936587c7e,933c9d4920a3eddc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:05.217 WARN 2458 --- [scanner-112187116-1,762910a936587c7e,bd6e4d946b8b3f16] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:05.707 INFO 2458 --- [prometheus-minute-schedule-1,6a95c7aae1b59530,e916dc6049a9d782] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:05.715 INFO 2458 --- [prometheus-minute-schedule-1,6a95c7aae1b59530,e916dc6049a9d782] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:26:06.054 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:26:06.103 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:26:06.128 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:26:06.128 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:26:06.128 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:26:06.128 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:26:06.132 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:26:06.133 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:06.134 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:26:06.135 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:06.136 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:26:06.397 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:06.398 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:06.604 WARN 2458 --- [ocp-async-3,b11512adb111047b,5ae97943f38321c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:06.604 INFO 2458 --- [ocp-async-1,84f03267b693fbd7,b36326c21ad930ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:06.737 INFO 2458 --- [ocp-schedules-22,2f28f066e9c0b63a,75fed1b366bd9941] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:07.055 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:23:30, start=2025-10-27T11:20:00, end=2025-10-27T11:22:30, updates=214, granule=120000ms, delay=210000ms 2025-10-27 11:26:07.060 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:23:30, target=2025-10-27T11:22:30, granule=120000ms, delay=60000ms) 2025-10-27 11:26:07.060 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:22:30, target=2025-10-27T11:12:30, granule=600000ms, delay=600000ms) 2025-10-27 11:26:07.060 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=1062ms, period=120000ms 2025-10-27 11:26:07.421 INFO 2458 --- [ocp-schedules-29,2afb99059bb0e6ce,8f590d5dce5a2d44] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:26:07.430 INFO 2458 --- [ocp-schedules-29,2afb99059bb0e6ce,8f590d5dce5a2d44] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:07.430 INFO 2458 --- [ocp-schedules-29,2afb99059bb0e6ce,8f590d5dce5a2d44] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:08.108 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:26:08.108+08:00, tenantSize=6 2025-10-27 11:26:08.135 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761535568108000 2025-10-27 11:26:08.135 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761535267984000 2025-10-27 11:26:08.191 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:26:08.207 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761535568195000 2025-10-27 11:26:08.207 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761535268070000 2025-10-27 11:26:08.247 INFO 2458 --- [ocp-schedules-28,a1016060f09b8d6c,648ef6c1d6c76f70] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:26:08.533 INFO 2458 --- [ocp-async-1,0f38198b972bd2c6,4d7306a3cefdbd36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:08.604 WARN 2458 --- [ocp-async-7,d81f47901d1977b7,def190fce4617e7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:08.604 INFO 2458 --- [ocp-async-6,75fe5e52fed3ee59,d54637ddc206bf4b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:08.707 INFO 2458 --- [prometheus-minute-schedule-1,8b919c6b7cf742ce,90ff7be37053f791] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:08.717 INFO 2458 --- [prometheus-minute-schedule-1,8b919c6b7cf742ce,90ff7be37053f791] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:09.379 INFO 2458 --- [ocp-schedules-14,a47fd29f0a70c097,1391a477a9089081] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:26:09.387 INFO 2458 --- [ocp-schedules-14,a47fd29f0a70c097,1391a477a9089081] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:09.398 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:09.398 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:09.475 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:26:09.476 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.480 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:26:09.489 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.489 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.517 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.518 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:26:09.518 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.518 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.521 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:26:09.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.543 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.545 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.545 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.573 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.573 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.573 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.601 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.601 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:26:09.604 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.605 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.631 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.632 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.632 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.633 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.634 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.635 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.635 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.636 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.637 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.637 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.637 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.638 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.638 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.639 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.639 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:26:09.642 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:26:09.665 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.665 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:26:09.668 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.668 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:26:09.690 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:09.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.692 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:26:09.699 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:26:09.702 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.702 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:26:09.702 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.702 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:26:09.705 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:26:09.705 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:26:09.730 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.730 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:26:09.733 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.734 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:26:09.766 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.767 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:26:09.794 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:26:09.846 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.847 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.848 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.849 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.849 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.850 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.851 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.852 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.853 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.854 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.854 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.855 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.856 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.857 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.857 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.858 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.859 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.860 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.860 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.861 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:26:09.861 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.863 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:26:09.863 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.864 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:26:09.864 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:26:09.867 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:26:09.953 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:09.953 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:09.953 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:09.954 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:09.954 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:09.954 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:09.954 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:26:09.981 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:26:09.985 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:26:09.985 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:26:09.985 INFO 2458 --- [ocp-schedules-14,a47fd29f0a70c097,1391a477a9089081] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:26:10.013 INFO 2458 --- [ocp-schedules-3,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:26:10.014 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:26:10.056 INFO 2458 --- [Check obproxy1,a9c9df7383b40131,e3a4de550f816559] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:26:10.056 INFO 2458 --- [Check obproxy1,a9c9df7383b40131,e3a4de550f816559] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:26:10.081 INFO 2458 --- [Collect obproxy1,2898a34890161bef,057fa04c6798fd0a] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:26:10.082 INFO 2458 --- [Collect obproxy1,2898a34890161bef,057fa04c6798fd0a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:26:10.091 INFO 2458 --- [Check obproxy1,a9c9df7383b40131,e3a4de550f816559] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:26:10.095 INFO 2458 --- [Check obproxy1,a9c9df7383b40131,e3a4de550f816559] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:26:10.095 INFO 2458 --- [Check obproxy1,a9c9df7383b40131,e3a4de550f816559] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:26:10.116 INFO 2458 --- [Collect obproxy1,2898a34890161bef,057fa04c6798fd0a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:26:10.119 INFO 2458 --- [Collect obproxy1,2898a34890161bef,057fa04c6798fd0a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:26:10.122 INFO 2458 --- [Collect obproxy1,2898a34890161bef,057fa04c6798fd0a] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:26:10.177 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:26:10.541 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=1000, affectRows=1000 2025-10-27 11:26:10.546 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=994, validCount=992 2025-10-27 11:26:10.607 INFO 2458 --- [ocp-async-6,2fdfee9d22f0760a,4b676bdbca62ff4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:10.607 WARN 2458 --- [ocp-async-8,52adc1c6c5ea1970,d345a754675fedad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:10.614 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=1000, affectRows=1000 2025-10-27 11:26:10.622 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=999, validCount=995 2025-10-27 11:26:10.638 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505658) Duplicate entry '835014129065861120-1761535440' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505658) Duplicate entry '835014129065861120-1761535440' for key 'PRIMARY' 2025-10-27 11:26:11.608 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535500, nodeCount=817, affectRows=817 2025-10-27 11:26:11.707 INFO 2458 --- [prometheus-minute-schedule-1,fb6b98048db3f4a0,b1805676ef5b2eb2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:11.716 INFO 2458 --- [prometheus-minute-schedule-1,fb6b98048db3f4a0,b1805676ef5b2eb2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:11.715991+08:00, lastProcessTime=2025-10-27T11:26:01.714994+08:00 2025-10-27 11:26:12.269 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:11.715991+08:00 2025-10-27 11:26:12.398 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:12.399 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:12.605 WARN 2458 --- [ocp-async-4,942eb8a80e825f33,72861cfe02bafd47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:12.605 INFO 2458 --- [ocp-async-6,c64ac7943e67f271,1356b16fd1eb2c84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:12.715 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=1000, affectRows=1000 2025-10-27 11:26:12.720 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=996, validCount=992 2025-10-27 11:26:12.781 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=1000, affectRows=1000 2025-10-27 11:26:12.784 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=996, validCount=992 2025-10-27 11:26:12.845 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=97, affectRows=97 2025-10-27 11:26:12.845 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=96, validCount=96 2025-10-27 11:26:14.605 WARN 2458 --- [ocp-async-6,cf430df2541bc7b4,a3982ad685456cda] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:14.605 INFO 2458 --- [ocp-async-7,5ef72be95262bdc4,c7542353db7e62ec] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:14.707 INFO 2458 --- [prometheus-minute-schedule-1,fd9218ad82ce779c,b30c5722aa8dc268] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:14.720 INFO 2458 --- [prometheus-minute-schedule-1,fd9218ad82ce779c,b30c5722aa8dc268] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:15.059 INFO 2458 --- [ocp-schedules-15,7412481db41bfa0a,edba6a5960d3324b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:15.063 INFO 2458 --- [ocp-schedules-15,7412481db41bfa0a,edba6a5960d3324b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:15.331 INFO 2458 --- [ocp-schedules-29,b8ba14900d8f1200,fd8ab395c7597d43] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:15.384 WARN 2458 --- [scanner-112188816-1,b8ba14900d8f1200,79b27f022852fb2d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:15.401 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:15.402 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:15.516 INFO 2458 --- [alarm-detect-7,b8ba14900d8f1200,046ffa938f28fb5b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535545 ~ 1761535565 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535560, value=57.407)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535560, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535560=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:26:15.521 INFO 2458 --- [alarm-detect-7,b8ba14900d8f1200,046ffa938f28fb5b] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:26:15.618 INFO 2458 --- [ocp-schedules-7,7692b4305fdc6809,71a83938de887fca] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:26:15.618+08:00, clusters=[1, 2] 2025-10-27 11:26:15.622 INFO 2458 --- [ocp-schedules-7,7692b4305fdc6809,71a83938de887fca] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:26:15.625 INFO 2458 --- [ocp-schedules-7,7692b4305fdc6809,71a83938de887fca] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:26:16.605 WARN 2458 --- [ocp-async-7,a496d7d156003189,3dacc145495a0987] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:16.605 INFO 2458 --- [ocp-async-8,12a6ff2037ec98f9,b3a01148bc58a931] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:16.744 INFO 2458 --- [ocp-schedules-28,0a60a3b4b2b6fdb3,9a0ac4940e5b916c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:17.075 INFO 2458 --- [ocp-schedules-24,c20fc1b7c41a8c2c,edb633c7a1abdc5b] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:26:17.075 INFO 2458 --- [ocp-schedules-24,c20fc1b7c41a8c2c,edb633c7a1abdc5b] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:26:17.142 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:26:17.142 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:26:17.142 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:26:17.143 INFO 2458 --- [Collect all tenants major compactions5,c20fc1b7c41a8c2c,dd90af9513324187] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:26:17.143 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:26:17.143 INFO 2458 --- [Collect all tenants major compactions6,c20fc1b7c41a8c2c,d1f078e998198f9a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:26:17.190 INFO 2458 --- [Collect all tenants major compactions5,c20fc1b7c41a8c2c,dd90af9513324187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.191 INFO 2458 --- [Collect all tenants major compactions5,c20fc1b7c41a8c2c,dd90af9513324187] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:26:17.195 INFO 2458 --- [Collect all tenants major compactions5,c20fc1b7c41a8c2c,dd90af9513324187] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:26:17.202 INFO 2458 --- [Collect all tenants major compactions6,c20fc1b7c41a8c2c,d1f078e998198f9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.204 INFO 2458 --- [Collect all tenants major compactions6,c20fc1b7c41a8c2c,d1f078e998198f9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:26:17.207 INFO 2458 --- [Collect all tenants major compactions6,c20fc1b7c41a8c2c,d1f078e998198f9a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:26:17.214 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.214 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.215 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.216 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.216 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.217 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:26:17.217 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:26:17.217 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.217 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.218 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.218 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.218 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.219 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:26:17.221 INFO 2458 --- [Collect all tenants major compactions4,c20fc1b7c41a8c2c,d2ab1b82b9723fbd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:26:17.221 INFO 2458 --- [Collect all tenants major compactions1,c20fc1b7c41a8c2c,30a28a497076a9c4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:26:17.223 INFO 2458 --- [Collect all tenants major compactions2,c20fc1b7c41a8c2c,7c025e894c59befc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:26:17.235 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.236 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.236 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:17.236 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:17.236 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:17.237 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:26:17.241 INFO 2458 --- [Collect all tenants major compactions3,c20fc1b7c41a8c2c,32c7a47f24a1e864] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:26:17.243 INFO 2458 --- [ocp-schedules-24,c20fc1b7c41a8c2c,edb633c7a1abdc5b] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:26:17.707 INFO 2458 --- [prometheus-minute-schedule-1,ea5f2aba9152910b,30c53b34daac0d66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:17.717 INFO 2458 --- [prometheus-minute-schedule-1,ea5f2aba9152910b,30c53b34daac0d66] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:17.890 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535500, nodeCount=1000, affectRows=1000 2025-10-27 11:26:17.894 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535500, nodeCount=1000, validCount=987 2025-10-27 11:26:18.402 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:18.403 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:18.533 INFO 2458 --- [ocp-async-8,c73a5d53c0ff8908,28d2de7d2a4ced3b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:18.605 INFO 2458 --- [ocp-async-5,4267cf8d343f3d6b,7429a2053f9ad858] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:18.605 WARN 2458 --- [ocp-async-1,be195a476d98e62a,329dd9ad21877291] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:18.848 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.858 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicSt 2025-10-27 11:26:18.860 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.861 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.862 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.864 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.868 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.870 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.873 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.877 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.882 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.886 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.891 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.895 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.896 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.897 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.898 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.899 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.901 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.902 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.903 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.904 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.905 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.906 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.907 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.908 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:20:18.243, end=2025-10-27T11:25:18.843} 2025-10-27 11:26:18.908 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:26:18.961 INFO 2458 --- [ocp-schedules-25,1a5a8f8ec9e4c3d5,16d3ccb86c1fa21d] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:18.961 INFO 2458 --- [ocp-schedules-25,1a5a8f8ec9e4c3d5,16d3ccb86c1fa21d] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:26:20.239 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235} 2025-10-27 11:26:20.248 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyT 2025-10-27 11:26:20.248 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.250 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.250 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:26:20.250 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.253 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.257 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.257 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:26:20.257 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.259 INFO 2458 --- [ocp-schedules-17,e44d332450c075fc,3d92fc3a67acad1d] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:26:20.360 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180360000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.360Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.412 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 260 2025-10-27 11:26:20.413 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.439 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.439 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.459 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.459 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.493 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.493 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.499 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:26:20.528 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.36, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.529 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.536 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 60 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.605 WARN 2458 --- [ocp-async-6,8d2ba97ca3246b43,15d43523db01b65a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:20.605 INFO 2458 --- [ocp-async-9,9f973c7df55f3e09,2aca1ad198f8a11e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:20.619 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:26:20.619 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.707 INFO 2458 --- [prometheus-minute-schedule-1,9e64de55a773f473,8abb7106779e977f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:20.726 INFO 2458 --- [prometheus-minute-schedule-1,9e64de55a773f473,8abb7106779e977f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:20.760 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180760000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.760Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.760 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180760000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=E22B2DD3069F288541753D6F7040B0AB, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.760Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.764 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180764000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.764Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.765 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180765000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=CD71762358D36B6FE730FCB2B03CC85B, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.765Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.767 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:26:20.767 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.768 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180768000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.768Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.768 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:26:20.768 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.769 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180769000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F2FF310469132459527053D6EC7D4B13, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.769Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.770 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449180770000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=26FF43D80B9F633E93B3C360FF8712F4, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:26:20.770Z, endTime=2025-10-27T03:24:20.235Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:26:20.774 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:26:20.774 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.775 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 2 2025-10-27 11:26:20.775 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.776 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:26:20.776 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.791 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 264 2025-10-27 11:26:20.791 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.793 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 97 2025-10-27 11:26:20.793 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:26:20.798 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.798 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.798 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.798 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E22B2DD3069F288541753D6F7040B0AB, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.801 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.801 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=CD71762358D36B6FE730FCB2B03CC85B, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.802 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.802 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F2FF310469132459527053D6EC7D4B13, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.803 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.803 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.803 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.803 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.805 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.805 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.806 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.806 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.813 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.813 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.813 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.813 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.824 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.824 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.826 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.826 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=26FF43D80B9F633E93B3C360FF8712F4, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.829 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.829 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.834 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.834 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E22B2DD3069F288541753D6F7040B0AB, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.835 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.835 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=CD71762358D36B6FE730FCB2B03CC85B, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.836 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:26:20.836 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:26:20.837 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:26:20.837 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.76, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=E22B2DD3069F288541753D6F7040B0AB, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.839 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:26:20.841 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.841 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.842 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.842 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F2FF310469132459527053D6EC7D4B13, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.843 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.843 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.846 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:26:20.846 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.768, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.846 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:26:20.846 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.769, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=F2FF310469132459527053D6EC7D4B13, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.847 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:26:20.847 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.76, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.854 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.854 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.855 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.765, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=CD71762358D36B6FE730FCB2B03CC85B, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.858 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:26:20.862 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:26:20.862 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535460235000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=26FF43D80B9F633E93B3C360FF8712F4, sqlIdList=null, dynamicSql=false) 2025-10-27 11:26:20.866 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:26:20.868 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.764, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.875 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:26:20.77, end=2025-10-27T11:24:20.235}, clusterId=2, tenantId=5, sqlId=26FF43D80B9F633E93B3C360FF8712F4, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:26:20.875 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.877 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.877 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:26:20.877 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.880 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:26:20.881 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.884 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:26:20.885 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:22:19.853, end=2025-10-27T11:24:20.235},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:26:20.885 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:26:20.885 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:26:20.887 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:26:21.057 INFO 2458 --- [ocp-schedules-7,b7d4614f25b66cef,cde803145dd3c841] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:26:21.057 INFO 2458 --- [ocp-schedules-7,b7d4614f25b66cef,cde803145dd3c841] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:26:21.064 INFO 2458 --- [ocp-schedules-7,b7d4614f25b66cef,cde803145dd3c841] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:26:21.064 INFO 2458 --- [ocp-schedules-7,b7d4614f25b66cef,cde803145dd3c841] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:26:21.176 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,66df7242093936bf,b5952f07e5f3b920] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=66df7242093936bf, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:26:21.176 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,66df7242093936bf,b5952f07e5f3b920] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=66df7242093936bf, duration=1 ms] 2025-10-27 11:26:21.403 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:21.404 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:21.407 INFO 2458 --- [ocp-schedules-10,3952209127964360,5b98cd7f01345e7c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:21.407 INFO 2458 --- [ocp-schedules-1,09f096f682922871,5c85d9d8ba1a48a0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:21.407 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:26:21.408 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:26:21.408 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:26:21.408 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:26:21.408 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:26:21.409 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:26:21.409 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:26:21.409 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:26:21.409 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:26:21.409 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:26:21.419 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:21.420 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:21.421 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:21.422 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:21.422 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:21.423 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:21.423 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:21.423 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:21.425 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:21.425 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:21.426 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:21.426 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:21.427 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:21.427 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:21.427 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:21.427 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:21.427 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:21.429 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:21.430 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:21.432 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:21.432 INFO 2458 --- [ocp-schedules-15,fcf40df7445ef5e7,7acedde2deabed92] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:21.433 INFO 2458 --- [host-check-clockdiff1,fcf40df7445ef5e7,e694a234311ce0c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:26:21.433 INFO 2458 --- [host-check-clockdiff2,fcf40df7445ef5e7,dd5b5ee8cfc61575] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:26:21.434 INFO 2458 --- [host-check-clockdiff3,fcf40df7445ef5e7,afd41c5aeda06380] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:26:21.434 INFO 2458 --- [host-check-clockdiff4,fcf40df7445ef5e7,4576ec9d39124514] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:26:21.435 INFO 2458 --- [host-check-clockdiff5,fcf40df7445ef5e7,0c00875119185c6b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:26:21.455 INFO 2458 --- [ocp-schedules-30,9f16d8362c270266,6ae5275480ab2a60] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:26:21.458 INFO 2458 --- [host-check-clockdiff3,fcf40df7445ef5e7,afd41c5aeda06380] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535581 -1 -1 , error= 2025-10-27 11:26:21.460 INFO 2458 --- [host-check-clockdiff4,fcf40df7445ef5e7,4576ec9d39124514] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535581 -1 -1 , error= 2025-10-27 11:26:21.460 INFO 2458 --- [host-check-clockdiff1,fcf40df7445ef5e7,e694a234311ce0c6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535581 0 0 , error= 2025-10-27 11:26:21.463 INFO 2458 --- [host-check-clockdiff5,fcf40df7445ef5e7,0c00875119185c6b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535581 1 1 , error= 2025-10-27 11:26:21.545 INFO 2458 --- [ocp-schedules-11,a9d5cac1338d6eb0,5682ebec63cbdafa] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:26:21.549 INFO 2458 --- [ocp-schedules-22,0430ffa24b0f875f,e4b8a8db2f65eaab] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:26:21.551 INFO 2458 --- [ocp-schedules-11,a9d5cac1338d6eb0,5682ebec63cbdafa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:21.554 INFO 2458 --- [ocp-schedules-26,0e5ceda652019f4d,8818666318aa1812] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:26:21.557 INFO 2458 --- [ocp-schedules-22,0430ffa24b0f875f,e4b8a8db2f65eaab] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:21.557 INFO 2458 --- [ocp-schedules-22,0430ffa24b0f875f,e4b8a8db2f65eaab] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:26:21.570 INFO 2458 --- [ocp-schedules-5,82e7a7534c5f4973,76d9bc616a0cdacc] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:26:21.571 INFO 2458 --- [ocp-schedules-26,0e5ceda652019f4d,8818666318aa1812] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:21.579 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:26:21.580 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:26:21.580 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:26:21.580 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:26:21.581 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:26:21.582 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:26:21.580799606+08:00 2025-10-27 11:26:21.582 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:26:21.582287113+08:00 2025-10-27 11:26:21.582 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:26:21.582+08:00 2025-10-27 11:26:21.582 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:26:21.582+08:00 2025-10-27 11:26:21.583 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:26:21.582329084+08:00 2025-10-27 11:26:21.584 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:26:21.584 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:26:21.584 INFO 2458 --- [host-check-agent-status2,09f096f682922871,d5fc3c708cdef5cf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:26:21.584 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:26:21.583+08:00 2025-10-27 11:26:21.585 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:26:21.585 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:26:21.585 INFO 2458 --- [host-check-agent-status1,09f096f682922871,0bff3deb7575f404] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:26:21.587 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:26:21.587 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:26:21.588 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:26:21.589 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:26:21.589 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:26:21.589762688+08:00 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status4,09f096f682922871,87d8dc45adc16e1c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-agent-status5,09f096f682922871,611a427493f856cc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:26:21.590 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:26:21.590+08:00 2025-10-27 11:26:21.591 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:26:21.591 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:26:21.591 INFO 2458 --- [host-check-agent-status3,09f096f682922871,a167bd943f873f91] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:26:21.593 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:26:21.582+08:00 and status AVAILABLE 2025-10-27 11:26:21.595 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:26:21.582+08:00 and status AVAILABLE 2025-10-27 11:26:21.595 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:26:21.583+08:00 and status AVAILABLE 2025-10-27 11:26:21.598 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:26:21.590+08:00 and status AVAILABLE 2025-10-27 11:26:21.599 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:26:21.602 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:26:21.602 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:26:21.605 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:26:21.610 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:26:21.611 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:26:21.610230781+08:00 2025-10-27 11:26:21.611 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:26:21.611+08:00 2025-10-27 11:26:21.619 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.620 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:21.621 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:26:21.623 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:26:21.611+08:00 and status AVAILABLE 2025-10-27 11:26:21.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.626 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:21.626 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:21.629 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:21.629 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:26:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:26:21.634 INFO 2458 --- [ocp-schedules-11,a9d5cac1338d6eb0,5682ebec63cbdafa] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:26:21.637 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:21.638 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:21.639 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:21.640 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:21.641 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:21.641 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:21.653 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:21.656 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:21.657 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.658 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:21.693 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:21.699 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:21.706 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:21.710 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:26:21.713 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:21.715583+08:00, lastProcessTime=2025-10-27T11:26:11.715991+08:00 2025-10-27 11:26:21.723 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:26:21.723 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:21.734 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:26:21.739 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:26:21.749 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:26:21.762 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:26:21.766 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:26:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.769 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:26:21.770 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:26:21.778 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:26:21.780 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:26:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.787 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:26:21.823 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:21.823 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:21.829 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:21.834 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:21.839 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:26:21.841 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:26:21.842 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:21.845 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:21.845 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.846 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.850 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:26:21.850 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:21.854 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:26:21.858 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:26:21.863 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:26:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:21.875 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:26:21.877 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:26:21.884 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:26:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:21.889 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:21.889 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:21.889 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:26:21.890 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.890 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:21.892 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.894 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:26:21.895 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:21.898 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:26:21.921 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:21.923 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:21.925 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:21.931 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:21.931 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:21.938 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:26:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:21.940 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:26:21.942 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:21.951 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:21.952 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:26:21.952 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:21.953 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:21.954 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:26:21.961 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:26:21.962 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:26:21.968 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:26:21.994 INFO 2458 --- [host-check-host-status4,3952209127964360,ae554e2eba1d0649] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:26:21.994 INFO 2458 --- [host-check-host-status5,3952209127964360,5d500a0beb22c043] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:26:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:22.003 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:22.005 INFO 2458 --- [host-check-host-status2,3952209127964360,ce82c2a70eb134ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:26:22.006 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:22.006 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:22.009 INFO 2458 --- [host-check-host-status1,3952209127964360,4949e49b02d8d08d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:26:22.016 INFO 2458 --- [host-check-host-status3,3952209127964360,544cbc2644121189] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:26:22.045 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:22.045 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:22.062 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:26:22.091 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:22.091 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:22.105 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:26:22.105 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.106 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:26:22.108 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.108 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:26:22.124 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:22.124 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:22.125 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:26:22.147 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:22.148 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.148 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:26:22.162 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:22.170 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:26:22.170 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:26:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:22.204 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:26:22.213 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:26:22.221 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:22.224 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:22.281 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:26:22.350 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:26:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:26:22.353 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.354 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:26:22.369 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:21.715583+08:00 2025-10-27 11:26:22.383 INFO 2458 --- [host-check-clockdiff2,fcf40df7445ef5e7,dd5b5ee8cfc61575] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535582 -1 -1 , error= 2025-10-27 11:26:22.391 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:26:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:22.437 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:22.437 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:26:22.485 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:26:22.485 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:26:22.486 INFO 2458 --- [ocp-schedules-26,0e5ceda652019f4d,8818666318aa1812] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:26:22.605 INFO 2458 --- [ocp-async-3,9d8c19fac9563cca,d251b7244cdbba52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:22.605 WARN 2458 --- [ocp-async-2,b944688ad1cad379,b608829eee859434] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:23.707 INFO 2458 --- [prometheus-minute-schedule-1,5bcc9cc50d731359,45e954d6ad586b8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:23.717 INFO 2458 --- [prometheus-minute-schedule-1,5bcc9cc50d731359,45e954d6ad586b8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:24.404 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:24.405 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:24.605 INFO 2458 --- [ocp-async-8,884b828d3a117b0e,e1e810dea62fe1b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:24.605 WARN 2458 --- [ocp-async-4,abfdc26a0cdb489b,21cea7dec5204f86] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:25.071 INFO 2458 --- [ocp-schedules-16,60b710fada7eb938,9df3cdb499e52010] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:25.074 INFO 2458 --- [ocp-schedules-16,60b710fada7eb938,9df3cdb499e52010] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:25.558 INFO 2458 --- [ocp-schedules-26,c5bf7408a1d39742,01ea43b86d4d07a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:25.612 WARN 2458 --- [scanner-112189063-1,c5bf7408a1d39742,e82ddbf1f37893b2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:25.724 INFO 2458 --- [alarm-detect-9,c5bf7408a1d39742,58901dae82a42b29] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535560 ~ 1761535575 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535575, value=52.657)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535575, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535575=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:26:26.605 INFO 2458 --- [ocp-async-10,95bb13ef88888925,110078336acd02f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:26.606 WARN 2458 --- [ocp-async-5,b430b23ca49d9c29,581adbe5f17bdff5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:26.707 INFO 2458 --- [prometheus-minute-schedule-1,2be7839e93146441,7019bd32413473c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:26.720 INFO 2458 --- [prometheus-minute-schedule-1,2be7839e93146441,7019bd32413473c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:26.755 INFO 2458 --- [ocp-schedules-28,b50c41d9efce8cc7,3d8b05420b19c462] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:27.405 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:27.406 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:27.433 INFO 2458 --- [ocp-schedules-15,da8f3112509d3499,621bf5ae3df3211f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:26:27.440 INFO 2458 --- [ocp-schedules-15,da8f3112509d3499,621bf5ae3df3211f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:27.440 INFO 2458 --- [ocp-schedules-15,da8f3112509d3499,621bf5ae3df3211f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:28.533 INFO 2458 --- [ocp-async-10,7f8c1f79f597f176,d2eebb4fa0f6e5ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:28.606 INFO 2458 --- [ocp-async-7,8908343b4703834c,9d4a21245307d23d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:28.606 WARN 2458 --- [ocp-async-3,3e900e7564070bb4,b118f222f67065a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:28.996 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:26:28.996 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:26:28.998 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:26:28.998 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:26:28.999 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:26:28.999 INFO 2458 --- [ocp-schedules-27,8f5bc6beb0276f45,d5ad3690481c941d] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:26:29.454 INFO 2458 --- [ocp-schedules-18,38be738d384422fc,8e2fec5f566b35ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:26:29.460 INFO 2458 --- [ocp-schedules-18,38be738d384422fc,8e2fec5f566b35ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:26:29.460 INFO 2458 --- [ocp-schedules-18,38be738d384422fc,8e2fec5f566b35ba] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:26:29.707 INFO 2458 --- [prometheus-minute-schedule-1,db723d8e4cf92f4c,30508156bf28adcf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:29.715 INFO 2458 --- [prometheus-minute-schedule-1,db723d8e4cf92f4c,30508156bf28adcf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:30.406 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:30.407 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:30.606 INFO 2458 --- [ocp-async-1,641de463fe066f70,2374e72fc810a58f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:30.606 WARN 2458 --- [ocp-async-8,4271f41734a65c10,20718c46d56e0a8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:31.714126+08:00, lastProcessTime=2025-10-27T11:26:21.715583+08:00 2025-10-27 11:26:31.917 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:31.714126+08:00 2025-10-27 11:26:32.606 INFO 2458 --- [ocp-async-6,e8691f3735b5e11c,6f772449bd95cd53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:32.606 WARN 2458 --- [ocp-async-10,4dfc440e76a3e48c,c32f0a5e7f49ac08] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:32.707 INFO 2458 --- [prometheus-minute-schedule-1,8d62ca919c40d3f1,bb66932b693ae413] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:32.715 INFO 2458 --- [prometheus-minute-schedule-1,8d62ca919c40d3f1,bb66932b693ae413] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:33.407 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:33.408 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:33.913 INFO 2458 --- [ocp-schedules-2,d851db62671efbdc,fb7e0a53fbbf8cce] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:26:33.913 INFO 2458 --- [ocp-schedules-2,d851db62671efbdc,fb7e0a53fbbf8cce] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:26:34.606 INFO 2458 --- [ocp-async-9,3a065f6cbf376eb8,e18202a9fc670419] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:34.606 WARN 2458 --- [ocp-async-2,6743d84e338b5b98,5bdd3d932699e4e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:35.082 INFO 2458 --- [ocp-schedules-3,7b1e3f45484740a2,9d79d4462665acf5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:35.086 INFO 2458 --- [ocp-schedules-3,7b1e3f45484740a2,9d79d4462665acf5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:35.644 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:26:35.644 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:26:35.648 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:26:35.648 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:26:35.648 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:26:35.648 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:26:35.652 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:26:35.652 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:26:35.652 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:26:35.652 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:26:35.674 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:26:35.679 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:26:35.679 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:26:35.679 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:26:35.679 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:26:35.679 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:26:35.684 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:26:35.707 INFO 2458 --- [prometheus-minute-schedule-1,7d61f224c2804528,4ca35123d17aafff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:35.716 INFO 2458 --- [prometheus-minute-schedule-1,7d61f224c2804528,4ca35123d17aafff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:35.734 INFO 2458 --- [ocp-schedules-18,2dd547433ad14959,6e578808b46999e8] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:35.747 WARN 2458 --- [scanner-112189236-1,2dd547433ad14959,5380767d0134868a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:36.095 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:26:36.408 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:36.409 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:36.494 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.842699536S 2025-10-27 11:26:36.494 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:26:36.602 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:26:36.602 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:26:36.606 INFO 2458 --- [ocp-async-7,aa620a96b4ff296c,b96131e6e17a8eca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:36.607 WARN 2458 --- [ocp-async-4,a4757efd70e6fa87,87598e1200fd4ac3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:36.641 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:26:36.693 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:26:36.700 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:26:36.761 INFO 2458 --- [ocp-schedules-11,aabd8ccd22a5faf0,3ea770c3af025db7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:36.815 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] o.s.boot.SpringApplication : Started application in 1.167 seconds (JVM running for 258065.242) 2025-10-27 11:26:36.815 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.167352563S 2025-10-27 11:26:37.164 INFO 2458 --- [ocp-schedules-4,0354d2ce1057baad,c3003573cf0408ae] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:26:37.708 INFO 2458 --- [prometheus-cache-monitor1,7ff375f8a8bdec03,f0da4b1fea943f27] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20387,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:26:37.708 INFO 2458 --- [prometheus-cache-monitor1,7ff375f8a8bdec03,f0da4b1fea943f27] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:26:38.533 INFO 2458 --- [ocp-async-7,1736cba3b00620eb,c5cb9217d9a81e07] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:38.606 INFO 2458 --- [ocp-async-5,dbf7ecc173606410,f22c81b68c4279a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:38.606 WARN 2458 --- [ocp-async-6,d67773a78fc9a458,a81979b9f7f72ea6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:38.707 INFO 2458 --- [prometheus-minute-schedule-1,6f02db6e91fae651,5c0ba2ab7ac5ddab] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:38.714 INFO 2458 --- [prometheus-minute-schedule-1,6f02db6e91fae651,5c0ba2ab7ac5ddab] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:38.742 INFO 2458 --- [ocp-schedules-13,8a0d449e06938edf,c9289e84541fcc4a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:26:38.749 INFO 2458 --- [ocp-schedules-13,8a0d449e06938edf,c9289e84541fcc4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:38.749 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:26:38.749 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:26:38.749 INFO 2458 --- [ocp-schedules-13,8a0d449e06938edf,c9289e84541fcc4a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:26:38.765 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:26:38.765 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:26:38.826 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:38.826 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:38.872 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:38.872 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:38.872 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:38.873 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:26:38.885 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:38.885 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:38.885 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:38.886 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:26:38.955 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:38.956 INFO 2458 --- [ocp-schedules-26,af44e6788fc65be2,7b876feb9c0d6efe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:26:39.196 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:39.196 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:26:39.196 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:26:39.196 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:26:39.196 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:26:39.196 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:26:39.208 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:39.208 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:39.209 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:39.209 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:39.211 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:39.212 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:39.213 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:39.213 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:39.214 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:39.214 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:39.266 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:26:39.269 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:26:39.269 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:26:39.269 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:26:39.270 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:26:39.272 INFO 2458 --- [ocp-schedules-14,ceeb175840eccc09,0b24c1d5aa7e11cc] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:26:39.409 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:39.410 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:40.013 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:26:40.013 INFO 2458 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:26:40.049 INFO 2458 --- [Check obproxy1,0603a2097ac4d253,b3995be2de6c3cde] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:26:40.050 INFO 2458 --- [Collect obproxy1,d07529d63165016b,19a6980ac58025da] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:26:40.052 INFO 2458 --- [Check obproxy1,0603a2097ac4d253,b3995be2de6c3cde] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:26:40.054 INFO 2458 --- [Collect obproxy1,d07529d63165016b,19a6980ac58025da] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:26:40.081 INFO 2458 --- [Check obproxy1,0603a2097ac4d253,b3995be2de6c3cde] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:26:40.081 INFO 2458 --- [Collect obproxy1,d07529d63165016b,19a6980ac58025da] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:26:40.084 INFO 2458 --- [Check obproxy1,0603a2097ac4d253,b3995be2de6c3cde] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:26:40.084 INFO 2458 --- [Check obproxy1,0603a2097ac4d253,b3995be2de6c3cde] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:26:40.086 INFO 2458 --- [Collect obproxy1,d07529d63165016b,19a6980ac58025da] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:26:40.089 INFO 2458 --- [Collect obproxy1,d07529d63165016b,19a6980ac58025da] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:26:40.606 WARN 2458 --- [ocp-async-9,4cfd3f44ecef4858,af6c68ae296b300b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:40.606 INFO 2458 --- [ocp-async-2,8c1c2e1f74427cbe,0445a9084ef94fe6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:41.213 INFO 2458 --- [ocp-schedules-24,fc1da50fdd6c5f93,9b4332919580fb80] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:26:41.214 INFO 2458 --- [ocp-schedules-24,fc1da50fdd6c5f93,9b4332919580fb80] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:26:41.707 INFO 2458 --- [prometheus-minute-schedule-1,7c22432a63910b99,27b7018bbb0eaf7a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:41.714 INFO 2458 --- [prometheus-minute-schedule-1,7c22432a63910b99,27b7018bbb0eaf7a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:41.713344+08:00, lastProcessTime=2025-10-27T11:26:31.714126+08:00 2025-10-27 11:26:41.936 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:41.713344+08:00 2025-10-27 11:26:42.410 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:42.410 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:42.606 INFO 2458 --- [ocp-async-4,ba2fb12ac38be8da,68cfc9356fee1bc2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:42.607 WARN 2458 --- [ocp-async-7,9fcfb0a8d6ddc2bc,05300f097089efe7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:44.210 INFO 2458 --- [ocp-schedules-21,06ebece656616bde,ac277ac0a89ce028] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:26:44.607 WARN 2458 --- [ocp-async-6,fd869ed8cafea25a,bef9668cefa51ccf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:44.607 INFO 2458 --- [ocp-async-4,d50f6cfe88b895a4,d67a5a75aad3a028] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:44.707 INFO 2458 --- [prometheus-minute-schedule-1,48761fdd063a2c69,5d220aa3be9b4675] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:44.715 INFO 2458 --- [prometheus-minute-schedule-1,48761fdd063a2c69,5d220aa3be9b4675] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:45.093 INFO 2458 --- [ocp-schedules-15,856e533f7c39e19a,f2af53562e9a25e4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:45.097 INFO 2458 --- [ocp-schedules-15,856e533f7c39e19a,f2af53562e9a25e4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:45.102 INFO 2458 --- [prometheus-second-schedule-1,042c17e6e8e37b23,cbf66f724969c60e] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:26:45.410 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:45.411 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:46.607 INFO 2458 --- [ocp-async-4,e7d5859f84ab7da1,d104a84979eb2bf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:46.607 WARN 2458 --- [ocp-async-5,c66ce9b189d8f51f,2adc6776ecb876c6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:46.632 INFO 2458 --- [ocp-schedules-18,2583a7b7835cd047,7ec6574961a17ae2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:46.727 WARN 2458 --- [scanner-112189480-1,2583a7b7835cd047,c4f7507d93b7f3ec] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:46.767 INFO 2458 --- [ocp-schedules-8,79f3827e0a92ca30,90d65e56c24c125f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:46.884 INFO 2458 --- [alarm-detect-18,2583a7b7835cd047,0b78d73784db548d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535575 ~ 1761535596 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535590, value=45.199)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535590, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535590=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:26:47.443 INFO 2458 --- [ocp-schedules-14,c1dc7c047de9e530,6fdf032f1916102c] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:26:47.453 INFO 2458 --- [ocp-schedules-14,c1dc7c047de9e530,6fdf032f1916102c] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:47.453 INFO 2458 --- [ocp-schedules-14,c1dc7c047de9e530,6fdf032f1916102c] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:26:47.707 INFO 2458 --- [prometheus-minute-schedule-1,41064f8226496ffc,4be94dc322515433] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:47.715 INFO 2458 --- [prometheus-minute-schedule-1,41064f8226496ffc,4be94dc322515433] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:48.411 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:48.412 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:48.534 INFO 2458 --- [ocp-async-4,8f9121debda3c7b1,954eb64c03061bbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:48.607 WARN 2458 --- [ocp-async-2,1ee1a476e6a7682d,83e4ae87d453dfdd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:48.607 INFO 2458 --- [ocp-async-2,08e8133824670ce6,46ebd8e210b0543e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:49.343 INFO 2458 --- [ocp-schedules-26,fb4d97d9d6e99a56,cca7c98c49be56d8] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:26:49.343+08:00, clusters=[1, 2] 2025-10-27 11:26:49.361 INFO 2458 --- [ocp-schedules-26,fb4d97d9d6e99a56,cca7c98c49be56d8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:26:49.364 INFO 2458 --- [ocp-schedules-26,fb4d97d9d6e99a56,cca7c98c49be56d8] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:26:50.607 WARN 2458 --- [ocp-async-2,dbd6ecd6e704a052,9b40e2e4a05ec7f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:50.607 INFO 2458 --- [ocp-async-1,2a39138c0e7eda3b,a6ee96d998f1b301] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:50.707 INFO 2458 --- [prometheus-minute-schedule-1,d04452852929c91e,63e8c3fbd173f893] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:50.717 INFO 2458 --- [prometheus-minute-schedule-1,d04452852929c91e,63e8c3fbd173f893] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:51.212 INFO 2458 --- [ocp-schedules-24,06db99587446acb2,19c337cd1d8fdce7] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:26:51.213 INFO 2458 --- [ocp-schedules-24,06db99587446acb2,19c337cd1d8fdce7] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:26:51.220 INFO 2458 --- [ocp-schedules-24,06db99587446acb2,19c337cd1d8fdce7] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:26:51.221 INFO 2458 --- [ocp-schedules-24,06db99587446acb2,19c337cd1d8fdce7] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:26:51.409 INFO 2458 --- [ocp-schedules-1,3dc6570870c55d9d,af55dd79e54dfacb] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:51.409 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:26:51.410 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:26:51.410 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:26:51.410 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:26:51.410 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:26:51.411 INFO 2458 --- [ocp-schedules-22,261b454246dc2a35,56907dd155c10d43] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:51.411 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:26:51.411 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:26:51.412 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:26:51.412 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:26:51.412 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:26:51.412 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:51.414 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:51.424 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:51.425 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:51.425 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:51.426 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:51.426 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:51.427 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:51.427 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:51.428 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:51.429 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:51.430 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:51.430 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:51.430 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:51.430 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:51.430 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:51.431 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:51.431 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:51.432 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:51.432 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:51.432 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:51.432 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:51.437 INFO 2458 --- [ocp-schedules-14,166571a2689c1a68,f4dfc5dedbaa8bff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:26:51.438 INFO 2458 --- [host-check-clockdiff2,166571a2689c1a68,063125d7170e0b3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:26:51.438 INFO 2458 --- [host-check-clockdiff1,166571a2689c1a68,c5b335097c879429] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:26:51.439 INFO 2458 --- [host-check-clockdiff3,166571a2689c1a68,12eb84a184174b24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:26:51.440 INFO 2458 --- [host-check-clockdiff4,166571a2689c1a68,e8399a6013fd7a9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:26:51.440 INFO 2458 --- [host-check-clockdiff5,166571a2689c1a68,70b861d59434b0d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:26:51.442 INFO 2458 --- [ocp-schedules-6,1443693b59e623db,3fbb27e81483938e] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:26:51.459 INFO 2458 --- [host-check-clockdiff3,166571a2689c1a68,12eb84a184174b24] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535611 -1 -1 , error= 2025-10-27 11:26:51.485 INFO 2458 --- [host-check-clockdiff1,166571a2689c1a68,c5b335097c879429] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535611 0 0 , error= 2025-10-27 11:26:51.487 INFO 2458 --- [host-check-clockdiff4,166571a2689c1a68,e8399a6013fd7a9b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535611 0 0 , error= 2025-10-27 11:26:51.524 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:26:51.524 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:26:51.525 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:26:51.525 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:26:51.525 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:26:51.525 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:26:51.525 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:26:51.526 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:26:51.525232218+08:00 2025-10-27 11:26:51.526 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:26:51.526438676+08:00 2025-10-27 11:26:51.526 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:26:51.526633386+08:00 2025-10-27 11:26:51.527 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:26:51.526+08:00 2025-10-27 11:26:51.527 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:26:51.526+08:00 2025-10-27 11:26:51.527 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:26:51.526842700+08:00 2025-10-27 11:26:51.527 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-agent-status5,3dc6570870c55d9d,33a6b242a781d3a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:26:51.526+08:00 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:26:51.527+08:00 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:26:51.528 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:26:51.529 INFO 2458 --- [host-check-agent-status4,3dc6570870c55d9d,c73dc5541d68e02f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:26:51.529 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:26:51.529 INFO 2458 --- [host-check-agent-status1,3dc6570870c55d9d,54fd3d24625a6bfd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:26:51.529 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:26:51.529 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:26:51.532 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:26:51.530918028+08:00 2025-10-27 11:26:51.532 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:26:51.532 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:26:51.532+08:00 2025-10-27 11:26:51.539 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:26:51.539 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:26:51.539 INFO 2458 --- [host-check-agent-status3,3dc6570870c55d9d,d9b29890884ffb15] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:26:51.540 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:26:51.540 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:26:51.540 INFO 2458 --- [host-check-agent-status2,3dc6570870c55d9d,5cd270ed3ea04ac5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:26:51.543 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:26:51.527+08:00 and status AVAILABLE 2025-10-27 11:26:51.544 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:26:51.526+08:00 and status AVAILABLE 2025-10-27 11:26:51.545 INFO 2458 --- [ocp-schedules-3,7a7117108a50eb08,a4c530c9b773ea5c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:26:51.545 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:26:51.532+08:00 and status AVAILABLE 2025-10-27 11:26:51.547 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:26:51.526+08:00 and status AVAILABLE 2025-10-27 11:26:51.548 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:26:51.526+08:00 and status AVAILABLE 2025-10-27 11:26:51.548 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:26:51.549 INFO 2458 --- [ocp-schedules-4,81e8ca53a87ecbbf,1bfca66a1a086b90] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:26:51.550 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:26:51.551 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:26:51.551 INFO 2458 --- [ocp-schedules-3,7a7117108a50eb08,a4c530c9b773ea5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:51.553 INFO 2458 --- [ocp-schedules-4,81e8ca53a87ecbbf,1bfca66a1a086b90] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:51.553 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:26:51.554 INFO 2458 --- [ocp-schedules-2,afb25925f5a3ded1,170ac32b8d6184a1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:26:51.554 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:26:51.554 INFO 2458 --- [ocp-schedules-4,81e8ca53a87ecbbf,1bfca66a1a086b90] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:26:51.558 INFO 2458 --- [ocp-schedules-2,afb25925f5a3ded1,170ac32b8d6184a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:26:51.567 INFO 2458 --- [ocp-schedules-13,1b0b4c4983886495,ee679a9cc62a79e9] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:26:51.568 INFO 2458 --- [ocp-schedules-19,6c35536b42486dec,701783d44733ca47] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:26:51.569 INFO 2458 --- [ocp-schedules-17,2859bf8837296291,93de410e1dddd644] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:26:51.574 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:51.574 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:26:51.576 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:26:51.577 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:26:51.577 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:51.577 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:51.578 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:26:51.579 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:26:51.580 INFO 2458 --- [ocp-schedules-17,2859bf8837296291,93de410e1dddd644] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:26:51.581 INFO 2458 --- [ocp-schedules-19,6c35536b42486dec,701783d44733ca47] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:26:51.581 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:51.581 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:26:51.609 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.610 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:51.611 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:26:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.621 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:51.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:26:51.627 INFO 2458 --- [ocp-schedules-3,7a7117108a50eb08,a4c530c9b773ea5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:26:51.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.640 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:51.646 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:51.649 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:51.650 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:51.652 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:51.660 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:26:51.663 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:26:51.667 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:26:51.671 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:26:51.679 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:26:51.683 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:26:51.698 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:26:51.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.700 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.701 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.701 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.702 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:51.702 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:26:51.707 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:26:51.710 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:26:51.714285+08:00, lastProcessTime=2025-10-27T11:26:41.713344+08:00 2025-10-27 11:26:51.715 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.716 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:26:51.731 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:26:51.739 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:26:51.751 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:51.756 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:51.765 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:51.769 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:51.773 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:26:51.780 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:26:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:51.781 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:26:51.781 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.782 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:51.784 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:26:51.788 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:51.790 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.791 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:26:51.791 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:26:51.792 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:51.813 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:26:51.816 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:26:51.816 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:26:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:51.828 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:26:51.834 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:26:51.848 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:51.849 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:51.850 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:26:51.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:51.856 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:51.864 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:26:51.865 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:51.866 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:26:51.877 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:26:51.883 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:51.883 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:26:51.884 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.885 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:51.886 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:26:51.890 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:26:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.898 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.899 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.899 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:26:51.900 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:26:51.901 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:51.901 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:51.901 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:51.902 INFO 2458 --- [host-check-host-status5,261b454246dc2a35,aa0f83b542148267] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:26:51.905 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:26:51.906 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:51.906 INFO 2458 --- [host-check-host-status4,261b454246dc2a35,73fc9625d0401ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:26:51.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:26:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:26:51.919 INFO 2458 --- [host-check-host-status2,261b454246dc2a35,99660a6d93068739] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:26:51.922 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:26:51.926 INFO 2458 --- [host-check-host-status3,261b454246dc2a35,86cbb5cc254bd100] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:26:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:51.950 INFO 2458 --- [host-check-host-status1,261b454246dc2a35,092253bfb75ea224] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:26:51.977 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:26:51.987 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:26:51.714285+08:00 2025-10-27 11:26:51.990 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:51.990 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:52.022 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:26:52.022 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:26:52.023 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:52.028 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.029 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:26:52.048 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:26:52.069 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:52.073 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.074 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:26:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:26:52.096 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:26:52.108 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:52.109 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:26:52.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:52.143 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:26:52.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.226 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:26:52.229 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:26:52.241 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:26:52.250 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:26:52.253 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:26:52.323 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:26:52.372 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:26:52.372 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.372 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.372 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.373 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:26:52.374 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.374 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.375 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.376 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:26:52.387 INFO 2458 --- [host-check-clockdiff2,166571a2689c1a68,063125d7170e0b3e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535612 -2 -2 , error= 2025-10-27 11:26:52.415 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:26:52.456 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:26:52.457 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:26:52.520 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:26:52.520 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:26:52.520 INFO 2458 --- [ocp-schedules-2,afb25925f5a3ded1,170ac32b8d6184a1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:26:52.607 WARN 2458 --- [ocp-async-9,77b51c29e4571cdc,bca99d7717e6ab3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:52.607 INFO 2458 --- [ocp-async-8,7ba548ec91740cc5,0893975699ed79e0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:26:53.707 INFO 2458 --- [prometheus-minute-schedule-1,9798ee5df27d64ed,bc89ac6a5a61f781] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:53.715 INFO 2458 --- [prometheus-minute-schedule-1,9798ee5df27d64ed,bc89ac6a5a61f781] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:53.970 INFO 2458 --- [host-check-clockdiff5,166571a2689c1a68,70b861d59434b0d9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535613 0 0 , error= 2025-10-27 11:26:54.414 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:54.415 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:54.607 WARN 2458 --- [ocp-async-7,4fa600a6cc0b4801,4f54b5dc8da46e51] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:54.607 INFO 2458 --- [ocp-async-2,9a2222596cbe1e42,7d2a758f71b052ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:55.106 INFO 2458 --- [ocp-schedules-14,507ffbc215318eb5,a5a3998c64cbae9b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:26:55.110 INFO 2458 --- [ocp-schedules-14,507ffbc215318eb5,a5a3998c64cbae9b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:26:55.323 INFO 2458 --- [ocp-schedules-30,d3a662cc048b1772,f5faf3163923fadb] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:26:56.608 WARN 2458 --- [ocp-async-5,57ad8f8903aedb75,244d399d87739eb1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:56.608 INFO 2458 --- [ocp-async-3,d15f8f26a400b557,7bb5c6dae52bf793] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:56.707 INFO 2458 --- [prometheus-minute-schedule-1,40a2c05948c9cfae,9a06438643a9185f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:56.714 INFO 2458 --- [prometheus-minute-schedule-1,40a2c05948c9cfae,9a06438643a9185f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:26:56.774 INFO 2458 --- [ocp-schedules-14,720834a1fed1b59e,5ed964a581b66475] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:26:56.891 INFO 2458 --- [ocp-schedules-7,0d52a4f5cc393fbb,7821355b21b0029b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:26:56.944 WARN 2458 --- [scanner-112189723-1,0d52a4f5cc393fbb,d1db2a4330f93f27] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:26:56.984 INFO 2458 --- [alarm-detect-3,0d52a4f5cc393fbb,14c5a763e7234506] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535546 ~ 1761535606 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535560, value=203894.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203894.56, values={1761535560=203894.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:26:56.985 INFO 2458 --- [alarm-detect-3,0d52a4f5cc393fbb,14c5a763e7234506] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:26:57.085 INFO 2458 --- [alarm-detect-13,0d52a4f5cc393fbb,ccf0b16c7b9580ff] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535546 ~ 1761535606 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535560, value=93.15)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.15, values={1761535560=93.15}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:26:57.086 INFO 2458 --- [alarm-detect-13,0d52a4f5cc393fbb,ccf0b16c7b9580ff] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:26:57.116 INFO 2458 --- [alarm-detect-1,0d52a4f5cc393fbb,418f8ed5beeeb7db] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535547 ~ 1761535607 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535560, value=203895.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203895.912, values={1761535560=203895.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:26:57.116 INFO 2458 --- [alarm-detect-1,0d52a4f5cc393fbb,418f8ed5beeeb7db] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:26:57.138 INFO 2458 --- [alarm-detect-3,0d52a4f5cc393fbb,6316697ae8d41715] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535590 ~ 1761535607 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535605, value=43.605)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535605, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535605=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:26:57.415 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:26:57.416 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:26:58.534 INFO 2458 --- [ocp-async-3,3753defdbe2b5a0e,6adc2ac6e83ac04b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:26:58.608 INFO 2458 --- [ocp-async-4,305c6d6dd2dc950f,1483de575089f728] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:26:58.608 WARN 2458 --- [ocp-async-9,29467f535e83912d,5b8a40434ce5c7aa] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:26:59.446 INFO 2458 --- [ocp-schedules-29,1046de729eba8cb4,a54d50c26cfdb91f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:26:59.707 INFO 2458 --- [prometheus-minute-schedule-1,03f495f5eac81294,676e3c2c2730270e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535620, interval=60 2025-10-27 11:26:59.714 INFO 2458 --- [prometheus-minute-schedule-1,03f495f5eac81294,676e3c2c2730270e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:00.416 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:00.417 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:00.608 INFO 2458 --- [ocp-async-7,e8083362ffb7f0e4,d64fe0066f9c25ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:00.608 WARN 2458 --- [ocp-async-8,785b14f9169cc793,dc72082c13c10764] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:01.713562+08:00, lastProcessTime=2025-10-27T11:26:51.714285+08:00 2025-10-27 11:27:01.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:01.713562+08:00 2025-10-27 11:27:02.608 INFO 2458 --- [ocp-async-5,5dcdf9fd2373271d,ea06a87ba21509b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:02.608 WARN 2458 --- [ocp-async-1,fbaaf7d96e92d08e,608815314d380c61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:02.665 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9bd58d36fdad04fe,0263a7504c4ef905] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=9bd58d36fdad04fe, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:27:02.665 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,9bd58d36fdad04fe,0263a7504c4ef905] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=9bd58d36fdad04fe, duration=2 ms] 2025-10-27 11:27:02.707 INFO 2458 --- [prometheus-minute-schedule-1,ad3d439fb1ac2f1c,f4b8675adb8c19a7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:02.716 INFO 2458 --- [prometheus-minute-schedule-1,ad3d439fb1ac2f1c,f4b8675adb8c19a7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:03.417 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:03.419 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:04.060 INFO 2458 --- [ocp-schedules-5,10b16d344271c4f7,3732dcc05968d21c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:27:04.067 INFO 2458 --- [ocp-schedules-5,10b16d344271c4f7,3732dcc05968d21c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:04.103 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:04.104 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:27:04.116 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:04.116 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:04.116 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:04.116 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:04.117 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:04.117 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:27:04.130 INFO 2458 --- [ocp-schedules-5,10b16d344271c4f7,3732dcc05968d21c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:27:04.608 WARN 2458 --- [ocp-async-4,b5d7a8a821fd3d40,5eb06f76f655031d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:04.608 INFO 2458 --- [ocp-async-9,dc55066a7329f89a,7216a7241025a141] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:05.117 INFO 2458 --- [ocp-schedules-22,73762d14002533a7,d36b9bf47379e1cf] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:05.120 INFO 2458 --- [ocp-schedules-22,73762d14002533a7,d36b9bf47379e1cf] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:05.707 INFO 2458 --- [prometheus-minute-schedule-1,c8cd44e0a84fb424,735602e1ed2aaae6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:05.715 INFO 2458 --- [prometheus-minute-schedule-1,c8cd44e0a84fb424,735602e1ed2aaae6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:06.419 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:06.420 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:06.608 INFO 2458 --- [ocp-async-10,1aca5ec36090b3b7,7d00c3c5874803ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:06.608 WARN 2458 --- [ocp-async-7,aa7ae1ec93032883,a982eab773804281] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:06.780 INFO 2458 --- [ocp-schedules-20,b13d2e9cab42a6ef,7d045e9be7312927] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:07.151 INFO 2458 --- [ocp-schedules-19,5c5269b08c9f4085,06f61af9e9566c8e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:07.180 WARN 2458 --- [scanner-112189966-1,5c5269b08c9f4085,3f7eb990db71697f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:07.455 INFO 2458 --- [ocp-schedules-1,48059a11cc888b57,1c3116d36261b0cc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:27:07.463 INFO 2458 --- [ocp-schedules-1,48059a11cc888b57,1c3116d36261b0cc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:07.463 INFO 2458 --- [ocp-schedules-1,48059a11cc888b57,1c3116d36261b0cc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:08.534 INFO 2458 --- [ocp-async-10,e1e2df6576106b04,414c63aa03de1faf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:08.608 WARN 2458 --- [ocp-async-2,cbc0c8f5e1ec6962,d5f3c0809a2c8bc8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:08.608 INFO 2458 --- [ocp-async-3,5451dda5178c4cbd,57d63ba8bb6b2d6e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:08.707 INFO 2458 --- [prometheus-minute-schedule-1,b4e7a58273dfe5e3,994c14c6f023ed9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:08.714 INFO 2458 --- [prometheus-minute-schedule-1,b4e7a58273dfe5e3,994c14c6f023ed9c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:09.420 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:09.421 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:09.986 INFO 2458 --- [ocp-schedules-23,23d29263cca3efb1,394d2f958b3bb18b] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:27:09.993 INFO 2458 --- [ocp-schedules-23,23d29263cca3efb1,394d2f958b3bb18b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:10.012 INFO 2458 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:27:10.014 INFO 2458 --- [ocp-schedules-10,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:27:10.053 INFO 2458 --- [Check obproxy1,09ab0cedb968caad,a5e6e2eb1d1f41b4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:27:10.053 INFO 2458 --- [Check obproxy1,09ab0cedb968caad,a5e6e2eb1d1f41b4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:27:10.069 INFO 2458 --- [Collect obproxy1,008d2ceadcd2a690,4dc33cd5bb3f9fd7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:27:10.069 INFO 2458 --- [Collect obproxy1,008d2ceadcd2a690,4dc33cd5bb3f9fd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:27:10.091 INFO 2458 --- [Check obproxy1,09ab0cedb968caad,a5e6e2eb1d1f41b4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:27:10.095 INFO 2458 --- [Check obproxy1,09ab0cedb968caad,a5e6e2eb1d1f41b4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:27:10.095 INFO 2458 --- [Check obproxy1,09ab0cedb968caad,a5e6e2eb1d1f41b4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:27:10.101 INFO 2458 --- [Collect obproxy1,008d2ceadcd2a690,4dc33cd5bb3f9fd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:27:10.104 INFO 2458 --- [Collect obproxy1,008d2ceadcd2a690,4dc33cd5bb3f9fd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:27:10.107 INFO 2458 --- [Collect obproxy1,008d2ceadcd2a690,4dc33cd5bb3f9fd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:27:10.126 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:27:10.127 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.131 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:27:10.143 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.143 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.171 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.172 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.186 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:27:10.190 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.191 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.191 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:27:10.191 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.191 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.191 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:27:10.203 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.203 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.205 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.206 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.211 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.211 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.212 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.212 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.235 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.236 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.242 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.242 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.242 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.243 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.262 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.262 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.270 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.270 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.271 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.272 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:27:10.274 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.274 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.274 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.275 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.294 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.294 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.295 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.296 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.297 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.298 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.298 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.299 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.300 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.300 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.301 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.302 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.302 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:27:10.303 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.303 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.306 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.306 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.309 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:27:10.333 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.333 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.333 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.334 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:27:10.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.336 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:27:10.364 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:10.364 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.365 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:27:10.369 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.369 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.369 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.370 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:27:10.376 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.376 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:27:10.376 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.376 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:27:10.379 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:27:10.379 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:27:10.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.405 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.406 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:27:10.408 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.408 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.408 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.409 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:27:10.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.435 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.437 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.437 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:27:10.463 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.463 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.464 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.464 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:27:10.531 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.531 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.532 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.532 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.533 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.533 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.533 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.538 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.538 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.539 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.540 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.540 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.540 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.541 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.541 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.541 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.541 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.542 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.542 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.543 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.543 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.544 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.544 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.544 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.545 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.545 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.546 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.546 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.547 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.547 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.547 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.547 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.548 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.548 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.548 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.553 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:27:10.553 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.553 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.554 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.554 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:27:10.555 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.555 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.555 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.556 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:27:10.556 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:27:10.559 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:27:10.609 INFO 2458 --- [ocp-async-4,7e8a1ef95d9e5279,f1a59c3f81fdfbc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:10.609 WARN 2458 --- [ocp-async-9,234f3be7194439cf,9812466c531af492] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:10.652 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:10.653 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:27:10.680 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:27:10.681 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:27:10.685 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:27:10.685 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:27:10.687 INFO 2458 --- [ocp-schedules-23,23d29263cca3efb1,394d2f958b3bb18b] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:27:11.707 INFO 2458 --- [prometheus-minute-schedule-1,6049537041078c4b,de0364faf313a049] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:11.714 INFO 2458 --- [prometheus-minute-schedule-1,6049537041078c4b,de0364faf313a049] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:11.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:11.713351+08:00, lastProcessTime=2025-10-27T11:27:01.713562+08:00 2025-10-27 11:27:11.776 INFO 2458 --- [metric-archive-async-7,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535560, nodeCount=821, affectRows=821 2025-10-27 11:27:11.917 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:11.713351+08:00 2025-10-27 11:27:12.037 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=1000, affectRows=1000 2025-10-27 11:27:12.041 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=994, validCount=992 2025-10-27 11:27:12.126 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=1000, affectRows=1000 2025-10-27 11:27:12.130 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=999, validCount=995 2025-10-27 11:27:12.145 ERROR 2458 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570103) Duplicate entry '835370603008958464-1761535500' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570103) Duplicate entry '835370603008958464-1761535500' for key 'PRIMARY' 2025-10-27 11:27:12.421 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:12.422 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:12.609 WARN 2458 --- [ocp-async-8,9af5e96a24719c64,ea0f4c33272bba22] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:12.609 INFO 2458 --- [ocp-async-8,5b05428b02d8ad00,16addc30b4e76ade] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:12.879 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=1000, affectRows=1000 2025-10-27 11:27:12.883 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=1000, validCount=996 2025-10-27 11:27:12.910 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=97, affectRows=97 2025-10-27 11:27:12.910 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=95, validCount=95 2025-10-27 11:27:12.921 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=1000, affectRows=1000 2025-10-27 11:27:12.925 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=996, validCount=992 2025-10-27 11:27:14.609 WARN 2458 --- [ocp-async-10,6fb01dc8426c2ceb,b649e38670a306f7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:14.609 INFO 2458 --- [ocp-async-5,68b1813b7cf7a0a5,8e95daf0b1645ef5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:14.707 INFO 2458 --- [prometheus-minute-schedule-1,45fd965069c97392,6c06d7cafba117ec] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:14.715 INFO 2458 --- [prometheus-minute-schedule-1,45fd965069c97392,6c06d7cafba117ec] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:15.042 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:27:15.042 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:27:15.051 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.052 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:27:15.055 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.056 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.057 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.058 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:27:15.067 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.067 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.068 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.069 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:27:15.075 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:27:15.075 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:27:15.076 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.077 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:27:15.079 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.079 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.080 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.081 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:27:15.102 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.103 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.103 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.104 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:27:15.124 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.125 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.126 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.126 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:27:15.127 INFO 2458 --- [ocp-schedules-18,421d85f2dabcb4a3,2c71a209671e5ad5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:15.132 INFO 2458 --- [ocp-schedules-18,421d85f2dabcb4a3,2c71a209671e5ad5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:15.143 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.144 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.145 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.145 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:27:15.159 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.160 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.161 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.161 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:27:15.180 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.180 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.181 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.182 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:27:15.196 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.196 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.197 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.198 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:27:15.211 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.212 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.212 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.213 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:27:15.220 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.221 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.222 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.222 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:27:15.229 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.230 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.231 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.231 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:27:15.237 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.237 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.238 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.239 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:27:15.246 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.247 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.247 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.248 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:27:15.257 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.258 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.259 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.259 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:27:15.267 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.268 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.269 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.269 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:27:15.276 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.277 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.278 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.278 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:27:15.285 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.285 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.286 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.298 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:27:15.305 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.306 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.306 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.307 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:27:15.315 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.316 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.317 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.317 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:27:15.329 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.330 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.331 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.332 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:27:15.342 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.343 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.344 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.344 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:27:15.357 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.357 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.358 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.359 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:27:15.365 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.365 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.366 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.367 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:27:15.374 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.374 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.375 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.376 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:27:15.381 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.382 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.383 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.383 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:27:15.389 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.389 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.390 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.390 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:27:15.396 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.396 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.397 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.397 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:27:15.404 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.404 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.405 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.405 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:27:15.412 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.412 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.413 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.414 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:27:15.420 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.420 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.421 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.422 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:27:15.423 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:15.424 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:15.427 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.428 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.429 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.429 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:27:15.434 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.435 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.436 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.436 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:27:15.443 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.444 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.444 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.445 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:27:15.450 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.451 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.452 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.452 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:27:15.458 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.459 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.459 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.460 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:27:15.466 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.466 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.467 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.468 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:27:15.474 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.475 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.476 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.476 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:27:15.483 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.483 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.484 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.484 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:27:15.490 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.491 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.492 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.492 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:27:15.498 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.499 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.500 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.500 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:27:15.506 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.507 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.508 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.509 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:27:15.516 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.516 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.517 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.518 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:27:15.523 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.524 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.525 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.525 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:27:15.534 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.535 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.536 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.536 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:27:15.544 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.544 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.545 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.546 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:27:15.551 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.551 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.552 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.552 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:27:15.558 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.558 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.559 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.560 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:27:15.566 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.567 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.567 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.568 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:27:15.573 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.574 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.574 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.575 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:27:15.580 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.580 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.581 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.582 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:27:15.587 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.588 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.589 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.589 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:27:15.595 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.596 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.596 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.597 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:27:15.603 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.603 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.604 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.605 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:27:15.610 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.610 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.611 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.612 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:27:15.617 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.617 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.618 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.619 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:27:15.624 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.624 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.625 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.626 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:27:15.630 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.630 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.631 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.631 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:27:15.636 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.637 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:27:15.638 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:15.638 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:27:15.642 INFO 2458 --- [ocp-schedules-24,0d9a48f804e4b341,a722321c5ea24f62] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:27:15.712 INFO 2458 --- [ocp-schedules-30,0e86a72e3e91ffcd,f0c69cf74998d283] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:27:15.712+08:00, clusters=[1, 2] 2025-10-27 11:27:15.715 INFO 2458 --- [ocp-schedules-30,0e86a72e3e91ffcd,f0c69cf74998d283] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:27:15.718 INFO 2458 --- [ocp-schedules-30,0e86a72e3e91ffcd,f0c69cf74998d283] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:27:16.208 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535560, nodeCount=1000, affectRows=1000 2025-10-27 11:27:16.212 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535560, nodeCount=996, validCount=983 2025-10-27 11:27:16.609 WARN 2458 --- [ocp-async-3,a01852cbaf53f2ae,5495f3a8418eafd3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:16.609 INFO 2458 --- [ocp-async-3,ecf63f9a2c9b0a47,0a3637aed5fe83ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:16.787 INFO 2458 --- [ocp-schedules-25,bfc80fcbe54b92bb,c842a9eae3209ff5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:17.243 INFO 2458 --- [ocp-schedules-2,05e28e6be1cee5fb,4fcb45180281829a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:27:17.243 INFO 2458 --- [ocp-schedules-2,05e28e6be1cee5fb,4fcb45180281829a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:27:17.261 INFO 2458 --- [ocp-schedules-30,87abd65d792c7ec7,7af4d3345aafaf54] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:17.271 WARN 2458 --- [scanner-112191669-1,87abd65d792c7ec7,9d7b6fef573e420c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:17.319 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:27:17.338 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:27:17.338 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:27:17.339 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:27:17.361 INFO 2458 --- [Collect all tenants major compactions5,05e28e6be1cee5fb,ec3243e02dd0caba] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:27:17.362 INFO 2458 --- [Collect all tenants major compactions6,05e28e6be1cee5fb,057d6f2243ff2f11] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:27:17.378 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.378 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.379 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.379 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.379 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.381 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:27:17.391 INFO 2458 --- [Collect all tenants major compactions1,05e28e6be1cee5fb,20315dcb98d3ccd0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:27:17.397 INFO 2458 --- [Collect all tenants major compactions5,05e28e6be1cee5fb,ec3243e02dd0caba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.400 INFO 2458 --- [Collect all tenants major compactions5,05e28e6be1cee5fb,ec3243e02dd0caba] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:27:17.401 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.401 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.402 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.402 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.402 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions6,05e28e6be1cee5fb,057d6f2243ff2f11] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.403 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.404 INFO 2458 --- [Collect all tenants major compactions5,05e28e6be1cee5fb,ec3243e02dd0caba] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:27:17.404 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:27:17.405 INFO 2458 --- [Collect all tenants major compactions6,05e28e6be1cee5fb,057d6f2243ff2f11] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:27:17.407 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:17.408 INFO 2458 --- [Collect all tenants major compactions6,05e28e6be1cee5fb,057d6f2243ff2f11] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:27:17.409 INFO 2458 --- [Collect all tenants major compactions2,05e28e6be1cee5fb,5561f7444b0db427] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:27:17.410 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:27:17.411 INFO 2458 --- [Collect all tenants major compactions3,05e28e6be1cee5fb,d0b6a5edf6d4c239] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:27:17.413 INFO 2458 --- [Collect all tenants major compactions4,05e28e6be1cee5fb,990e425870c2d783] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:27:17.440 INFO 2458 --- [ocp-schedules-2,05e28e6be1cee5fb,4fcb45180281829a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:27:17.446 INFO 2458 --- [alarm-detect-19,87abd65d792c7ec7,0e8021de7f23f34b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535605 ~ 1761535627 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535620, value=51.681)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535620, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535620=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:27:17.707 INFO 2458 --- [prometheus-minute-schedule-1,7205ed137939bd25,50fe71a1fd78c9e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:17.714 INFO 2458 --- [prometheus-minute-schedule-1,7205ed137939bd25,50fe71a1fd78c9e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:18.424 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:18.425 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:18.534 INFO 2458 --- [ocp-async-3,61ac9b1a928c5c52,c7f2cc4882ce36c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:18.609 WARN 2458 --- [ocp-async-7,c2bc1454c5ded89f,ab2bb5ccee522fc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:18.609 INFO 2458 --- [ocp-async-1,1d01872988d990b5,807fb4209ff0a44a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:19.534 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,95fa86b6e31ff3bc,b31ded685b7419fd] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=95fa86b6e31ff3bc, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:27:19.534 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,95fa86b6e31ff3bc,b31ded685b7419fd] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=95fa86b6e31ff3bc, duration=1 ms] 2025-10-27 11:27:20.313 INFO 2458 --- [ocp-schedules-27,6ba07dea6a7f2bd4,b94cea47a8df2904] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:27:20.609 INFO 2458 --- [ocp-async-10,8a267ed20549962b,9c8b1b2001ddfc88] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:20.609 WARN 2458 --- [ocp-async-5,4840f37c297e3079,89fadbc0884d3efb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:20.707 INFO 2458 --- [prometheus-minute-schedule-1,cf9026054678c7ca,ad13c7e5b7a1d1c8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:20.714 INFO 2458 --- [prometheus-minute-schedule-1,cf9026054678c7ca,ad13c7e5b7a1d1c8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:21.376 INFO 2458 --- [ocp-schedules-9,550f675c493567b7,90b391f5d90949c8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:27:21.376 INFO 2458 --- [ocp-schedules-9,550f675c493567b7,90b391f5d90949c8] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:27:21.383 INFO 2458 --- [ocp-schedules-9,550f675c493567b7,90b391f5d90949c8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:27:21.383 INFO 2458 --- [ocp-schedules-9,550f675c493567b7,90b391f5d90949c8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:27:21.408 INFO 2458 --- [ocp-schedules-16,9468211d0148f425,466207080a2c3ef3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:21.408 INFO 2458 --- [ocp-schedules-10,4131d8671322f93d,4d054669dd56ecf0] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:27:21.409 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:27:21.410 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:27:21.410 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:27:21.410 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:27:21.410 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:27:21.425 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:21.425 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:21.426 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:21.426 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:21.426 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:21.426 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:21.427 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:21.427 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:21.427 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:21.430 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:21.431 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:21.432 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:21.432 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:21.433 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:21.433 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:21.433 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:21.433 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:21.434 INFO 2458 --- [ocp-schedules-7,2944d9bda490a673,01225ed494ecd226] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:21.434 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:21.470 INFO 2458 --- [ocp-schedules-13,193ba8fd1d26277e,bee6c6295847f1b0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:27:21.472 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:21.476 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:21.476 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:21.478 INFO 2458 --- [host-check-clockdiff2,2944d9bda490a673,737cd798ab0a10d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:27:21.478 INFO 2458 --- [host-check-clockdiff3,2944d9bda490a673,3d9f472200ec7ea1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:27:21.479 INFO 2458 --- [host-check-clockdiff4,2944d9bda490a673,b3c7ffeabd4439f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:27:21.480 INFO 2458 --- [host-check-clockdiff5,2944d9bda490a673,37dafb5380bc559c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:27:21.485 INFO 2458 --- [host-check-clockdiff1,2944d9bda490a673,2301700602935753] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:27:21.486 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:21.495 INFO 2458 --- [host-check-clockdiff5,2944d9bda490a673,37dafb5380bc559c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535641 0 0 , error= 2025-10-27 11:27:21.507 INFO 2458 --- [host-check-clockdiff1,2944d9bda490a673,2301700602935753] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535641 0 0 , error= 2025-10-27 11:27:21.513 INFO 2458 --- [host-check-clockdiff4,2944d9bda490a673,b3c7ffeabd4439f1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535641 0 0 , error= 2025-10-27 11:27:21.517 INFO 2458 --- [host-check-clockdiff3,2944d9bda490a673,3d9f472200ec7ea1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535641 -1 -1 , error= 2025-10-27 11:27:21.545 INFO 2458 --- [ocp-schedules-5,be8cddff2c92d5c2,f5ff2cc7b267d234] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:27:21.547 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:27:21.547 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:27:21.549 INFO 2458 --- [ocp-schedules-15,c2324bc2c0640010,cec130eb0aec1610] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:27:21.550 INFO 2458 --- [ocp-schedules-5,be8cddff2c92d5c2,f5ff2cc7b267d234] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:21.551 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:27:21.551 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:27:21.551197133+08:00 2025-10-27 11:27:21.552 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:27:21.552 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:27:21.551+08:00 2025-10-27 11:27:21.553 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:27:21.553 INFO 2458 --- [ocp-schedules-15,c2324bc2c0640010,cec130eb0aec1610] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:21.553 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:27:21.552364248+08:00 2025-10-27 11:27:21.554 INFO 2458 --- [ocp-schedules-17,b582c087bfb6b2f0,8b1bd0d59d1b3882] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:27:21.554 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:27:21.553+08:00 2025-10-27 11:27:21.555 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:27:21.555 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:27:21.555 INFO 2458 --- [host-check-agent-status4,4131d8671322f93d,415277e65277e4ce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:27:21.556 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:27:21.556 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:27:21.556 INFO 2458 --- [host-check-agent-status1,4131d8671322f93d,652dc561e70eec54] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:27:21.556 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:27:21.555458402+08:00 2025-10-27 11:27:21.557 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:27:21.556+08:00 2025-10-27 11:27:21.557 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:27:21.557 INFO 2458 --- [ocp-schedules-15,c2324bc2c0640010,cec130eb0aec1610] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:27:21.559 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:27:21.559063418+08:00 2025-10-27 11:27:21.559 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:27:21.559+08:00 2025-10-27 11:27:21.560 INFO 2458 --- [ocp-schedules-17,b582c087bfb6b2f0,8b1bd0d59d1b3882] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:21.560 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:27:21.561 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:27:21.551+08:00 and status AVAILABLE 2025-10-27 11:27:21.561 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:27:21.561 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:27:21.562 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:27:21.561505657+08:00 2025-10-27 11:27:21.563 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:27:21.562+08:00 2025-10-27 11:27:21.564 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:27:21.564 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:27:21.564 INFO 2458 --- [host-check-agent-status5,4131d8671322f93d,2c4cc8d59505f823] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:27:21.564 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:27:21.556+08:00 and status AVAILABLE 2025-10-27 11:27:21.565 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:27:21.553+08:00 and status AVAILABLE 2025-10-27 11:27:21.565 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:27:21.565 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:27:21.565 INFO 2458 --- [host-check-agent-status3,4131d8671322f93d,2a4f7d13bc2f8135] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:27:21.566 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:27:21.568 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:27:21.569 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:27:21.559+08:00 and status AVAILABLE 2025-10-27 11:27:21.570 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:27:21.571 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:27:21.571 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:27:21.571 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:27:21.571 INFO 2458 --- [host-check-agent-status2,4131d8671322f93d,227c0db6f28fa1fb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:27:21.572 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:27:21.562+08:00 and status AVAILABLE 2025-10-27 11:27:21.573 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:27:21.576 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:27:21.578 INFO 2458 --- [ocp-schedules-4,3b13b419ecf97756,a7d2b49cc5f6437d] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:27:21.589 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:21.590 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:21.592 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:21.593 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:21.596 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:21.596 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:21.598 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:21.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.599 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:21.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:21.599 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:21.600 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:27:21.601 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:27:21.621 INFO 2458 --- [ocp-schedules-5,be8cddff2c92d5c2,f5ff2cc7b267d234] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:27:21.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:21.655 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:21.658 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:21.662 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:21.670 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:21.671 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:27:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.672 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.676 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.679 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.679 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:21.679 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:27:21.679 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:21.689 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:27:21.690 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:21.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.691 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:27:21.696 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:27:21.702 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:27:21.702 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:27:21.712 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:27:21.722 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:27:21.723 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:21.722351+08:00, lastProcessTime=2025-10-27T11:27:11.713351+08:00 2025-10-27 11:27:21.728 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:27:21.736 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:27:21.742 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:21.747 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:21.748 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:21.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.750 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:21.751 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.754 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:27:21.755 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:21.757 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:21.757 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:27:21.758 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:21.764 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:27:21.766 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:21.766 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.766 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.767 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.767 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.767 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.768 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:27:21.775 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:21.779 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:27:21.781 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:21.782 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:21.787 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:27:21.790 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:27:21.797 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:27:21.797 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:27:21.808 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:21.817 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:27:21.821 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:21.822 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:27:21.828 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:21.830 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:27:21.834 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:21.834 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.835 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:21.836 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:27:21.836 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:27:21.846 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:27:21.846 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:27:21.848 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:21.849 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:21.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:21.860 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:21.863 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:21.869 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:27:21.870 INFO 2458 --- [host-check-host-status4,9468211d0148f425,c52a50ecdd251f18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:27:21.874 INFO 2458 --- [host-check-host-status5,9468211d0148f425,4df8360f40cd0ad8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:27:21.875 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.877 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.881 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:27:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:27:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:21.887 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:27:21.903 INFO 2458 --- [host-check-host-status3,9468211d0148f425,f1d307631d72cc43] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:27:21.920 INFO 2458 --- [host-check-host-status1,9468211d0148f425,06e911b42186d0ab] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:27:21.927 INFO 2458 --- [host-check-host-status2,9468211d0148f425,5f773eb6fc43ac1d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:27:21.936 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:21.936 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:21.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:27:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:21.974 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:21.981 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:27:21.982 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.983 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:27:21.985 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:21.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:27:21.996 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:21.722351+08:00 2025-10-27 11:27:22.002 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:27:22.010 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:22.010 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:22.026 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:22.027 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:22.027 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:27:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:22.051 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:27:22.051 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:27:22.081 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:22.081 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:22.116 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:22.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:27:22.169 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:27:22.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:22.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:22.231 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:27:22.279 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:27:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.279 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.280 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:22.281 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:27:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.283 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:22.284 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:27:22.323 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:27:22.404 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:27:22.404 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:27:22.404 INFO 2458 --- [ocp-schedules-17,b582c087bfb6b2f0,8b1bd0d59d1b3882] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:27:22.609 WARN 2458 --- [ocp-async-9,0cdb32c81124f868,b9c56f4523c44e8a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:22.609 INFO 2458 --- [ocp-async-3,c15a4dc85caa32d1,ee3f98a9499b3e40] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:23.268 INFO 2458 --- [host-check-clockdiff2,2944d9bda490a673,737cd798ab0a10d7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535643 -1 -1 , error= 2025-10-27 11:27:23.707 INFO 2458 --- [prometheus-minute-schedule-1,b4bc010b73199692,de41eebff438d794] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:23.715 INFO 2458 --- [prometheus-minute-schedule-1,b4bc010b73199692,de41eebff438d794] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:24.427 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:24.428 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:24.609 WARN 2458 --- [ocp-async-4,35529070e76dd3b6,b7a1defc4ffc6571] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:24.610 INFO 2458 --- [ocp-async-4,00bc12cccc3fc9fb,18a23d3b725ec195] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:25.140 INFO 2458 --- [ocp-schedules-26,063a16fc3dbaf29c,ec039bc33ff4ee59] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:25.145 INFO 2458 --- [ocp-schedules-26,063a16fc3dbaf29c,ec039bc33ff4ee59] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:26.610 WARN 2458 --- [ocp-async-1,526667243e30b7c1,51d67155a8cec4ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:26.610 INFO 2458 --- [ocp-async-5,7e216428a6cca935,3718827e42861377] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:26.707 INFO 2458 --- [prometheus-minute-schedule-1,ebb77d4880c7e02e,be4c3c844ed8282f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:26.715 INFO 2458 --- [prometheus-minute-schedule-1,ebb77d4880c7e02e,be4c3c844ed8282f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:26.795 INFO 2458 --- [ocp-schedules-14,a7a53e0209c9779a,ccaf356738189b3f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:27.428 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:27.429 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:27.453 INFO 2458 --- [ocp-schedules-11,65b7d1e4034833b4,89ec766f2d86ac4d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:27.465 INFO 2458 --- [ocp-schedules-20,6b78c15146c22a28,22cd1823bba99f74] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:27:27.478 INFO 2458 --- [ocp-schedules-20,6b78c15146c22a28,22cd1823bba99f74] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:27.478 INFO 2458 --- [ocp-schedules-20,6b78c15146c22a28,22cd1823bba99f74] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:27.489 WARN 2458 --- [scanner-112191935-1,65b7d1e4034833b4,0e4cef68724893fd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:27.636 INFO 2458 --- [alarm-detect-9,65b7d1e4034833b4,321f28f3663f307c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535620 ~ 1761535637 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535635, value=53.968)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535635, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535635=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:27:28.534 INFO 2458 --- [ocp-async-5,6a9ffa6cbd3477f7,3d3b756ddf1aeae4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:28.610 WARN 2458 --- [ocp-async-9,94119a29d0ae6c69,a3236e65369ed89a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:28.610 INFO 2458 --- [ocp-async-3,de83272a57ccf1a2,5c7bbf32f016ebf4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:29.000 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:27:29.000 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:27:29.006 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:27:29.006 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:27:29.011 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:27:29.011 INFO 2458 --- [ocp-schedules-1,dd1bb1753a04d793,67807987e9d6afd3] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:27:29.460 INFO 2458 --- [ocp-schedules-24,761ea257e74240e1,cb74d192129e487d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:27:29.465 INFO 2458 --- [ocp-schedules-24,761ea257e74240e1,cb74d192129e487d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:27:29.465 INFO 2458 --- [ocp-schedules-24,761ea257e74240e1,cb74d192129e487d] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:27:29.707 INFO 2458 --- [prometheus-minute-schedule-1,fecb1b2197871f63,46abe5ac0f00ff44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:29.715 INFO 2458 --- [prometheus-minute-schedule-1,fecb1b2197871f63,46abe5ac0f00ff44] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:30.429 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:30.431 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:30.610 WARN 2458 --- [ocp-async-8,86bc81dbcf27df9a,166fdabd1de92bb7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:30.611 INFO 2458 --- [ocp-async-4,2e5634ade4a50837,d09a61983dd10236] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:31.713326+08:00, lastProcessTime=2025-10-27T11:27:21.722351+08:00 2025-10-27 11:27:31.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:31.713326+08:00 2025-10-27 11:27:32.610 WARN 2458 --- [ocp-async-2,aec5d0ac1588a743,76d79496247c48d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:32.611 INFO 2458 --- [ocp-async-5,8c9564af013c90f6,964b93abbd1bc9d2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:32.707 INFO 2458 --- [prometheus-minute-schedule-1,4dbe8efc5852575a,cba5d4e60fa11438] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:32.718 INFO 2458 --- [prometheus-minute-schedule-1,4dbe8efc5852575a,cba5d4e60fa11438] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:33.431 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:33.432 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:33.923 INFO 2458 --- [ocp-schedules-20,5e269edaa8603c85,c4de5aebbc7ef450] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:27:33.923 INFO 2458 --- [ocp-schedules-20,5e269edaa8603c85,c4de5aebbc7ef450] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:27:34.372 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:27:34.372 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:27:34.392 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:27:34.392 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:27:34.392 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:27:34.404 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:27:34.404 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:27:34.404 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:27:34.418 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:27:34.418 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:27:34.418 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:27:34.429 INFO 2458 --- [ocp-schedules-30,7cfc591e1f55c1a7,11ef4a937897f483] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:27:34.610 WARN 2458 --- [ocp-async-9,6fd33e8f364b3b3e,d627b336095fdcb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:34.611 INFO 2458 --- [ocp-async-6,52e8f1b01f6ab93d,903606f7d7af3551] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:35.154 INFO 2458 --- [ocp-schedules-16,16fc53d1bfbab7af,f9eda46656a20f64] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:35.158 INFO 2458 --- [ocp-schedules-16,16fc53d1bfbab7af,f9eda46656a20f64] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:35.707 INFO 2458 --- [prometheus-minute-schedule-1,7954ac05ab171e50,8230ce9f5214dc2d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:35.716 INFO 2458 --- [prometheus-minute-schedule-1,7954ac05ab171e50,8230ce9f5214dc2d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:36.432 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:36.433 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:36.610 WARN 2458 --- [ocp-async-8,d33c6abb21758999,3c2eca6c4843f084] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:36.611 INFO 2458 --- [ocp-async-7,147528896af3bb48,5986f782dfc22797] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:36.802 INFO 2458 --- [ocp-schedules-2,2d9a0a6afc442718,79d4d51ea49964a9] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:37.645 INFO 2458 --- [ocp-schedules-15,1d33050a113af569,f90cdfea9ac303ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:37.664 WARN 2458 --- [scanner-112192100-1,1d33050a113af569,59b5771965e3dc22] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:37.708 INFO 2458 --- [prometheus-cache-monitor1,78f57d71367e121c,0c1c5654d22eda69] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20387,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:27:37.708 INFO 2458 --- [prometheus-cache-monitor1,78f57d71367e121c,0c1c5654d22eda69] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:27:38.534 INFO 2458 --- [ocp-async-7,381db87c7d688904,dee19a23adc4bbf7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:38.611 WARN 2458 --- [ocp-async-1,ffc11a24c3eec032,179ba14e5bc82f1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:38.611 INFO 2458 --- [ocp-async-5,c64fd089cdfce313,62a0c65dd5c5e1ef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:38.707 INFO 2458 --- [prometheus-minute-schedule-1,7295f42cc05fea57,c5f3f5344177ef54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:38.715 INFO 2458 --- [prometheus-minute-schedule-1,7295f42cc05fea57,c5f3f5344177ef54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:38.750 INFO 2458 --- [ocp-schedules-26,957914af2645619e,779873e5df193176] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:27:38.757 INFO 2458 --- [ocp-schedules-26,957914af2645619e,779873e5df193176] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:38.757 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:27:38.757 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:27:38.757 INFO 2458 --- [ocp-schedules-26,957914af2645619e,779873e5df193176] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:27:38.961 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:27:38.961 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:27:39.026 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:39.026 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:39.064 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:39.064 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:39.065 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:39.066 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:27:39.079 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:39.079 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:39.080 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:39.080 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:27:39.139 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:39.140 INFO 2458 --- [ocp-schedules-4,a569003df2cd4f23,082f4e482eec4095] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:27:39.281 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:39.281 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:27:39.282 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:27:39.282 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:27:39.282 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:27:39.282 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:27:39.293 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:39.294 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:39.294 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:39.294 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:39.295 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:39.296 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:39.297 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:39.298 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:39.298 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:39.299 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:39.351 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:27:39.352 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:27:39.353 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:27:39.357 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:27:39.357 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:27:39.359 INFO 2458 --- [ocp-schedules-22,be201270d2d25ae9,0784ea7547c2ebd0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:27:39.433 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:39.435 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:40.019 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:27:40.026 INFO 2458 --- [ocp-schedules-2,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:27:40.060 INFO 2458 --- [Check obproxy1,3649721bf2b8525a,3c26f9fc820245bb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:27:40.060 INFO 2458 --- [Check obproxy1,3649721bf2b8525a,3c26f9fc820245bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:27:40.063 INFO 2458 --- [Collect obproxy1,fc50ece0309db98e,51569dfc9c296e50] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:27:40.063 INFO 2458 --- [Collect obproxy1,fc50ece0309db98e,51569dfc9c296e50] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:27:40.092 INFO 2458 --- [Check obproxy1,3649721bf2b8525a,3c26f9fc820245bb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:27:40.093 INFO 2458 --- [Collect obproxy1,fc50ece0309db98e,51569dfc9c296e50] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:27:40.096 INFO 2458 --- [Check obproxy1,3649721bf2b8525a,3c26f9fc820245bb] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:27:40.096 INFO 2458 --- [Check obproxy1,3649721bf2b8525a,3c26f9fc820245bb] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:27:40.097 INFO 2458 --- [Collect obproxy1,fc50ece0309db98e,51569dfc9c296e50] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:27:40.099 INFO 2458 --- [Collect obproxy1,fc50ece0309db98e,51569dfc9c296e50] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:27:40.611 WARN 2458 --- [ocp-async-10,e94a574cdf043c0f,65704af5b52fb807] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:40.611 INFO 2458 --- [ocp-async-6,b35b8654b29f2217,df9725a6cdc94013] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:41.236 INFO 2458 --- [ocp-schedules-17,c04a0cb0913b98aa,7714afb572461acd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:27:41.237 INFO 2458 --- [ocp-schedules-17,c04a0cb0913b98aa,7714afb572461acd] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:27:41.707 INFO 2458 --- [prometheus-minute-schedule-1,1f052fb5aff12bc7,b521f09eea633742] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:41.713 INFO 2458 --- [prometheus-minute-schedule-1,1f052fb5aff12bc7,b521f09eea633742] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:41.712686+08:00, lastProcessTime=2025-10-27T11:27:31.713326+08:00 2025-10-27 11:27:41.910 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:41.712686+08:00 2025-10-27 11:27:42.435 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:42.436 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:42.611 WARN 2458 --- [ocp-async-3,5b905818f4a21e60,140f50e813b8a2eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:42.611 INFO 2458 --- [ocp-async-7,6f9c4e3cd44f3bd6,4ec89870af36259d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:44.218 INFO 2458 --- [ocp-schedules-7,730f5d6a5634ce7f,dc750af618a2f8d2] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:27:44.611 WARN 2458 --- [ocp-async-4,40629119383fd7e4,4cc096b3d1660161] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:44.611 INFO 2458 --- [ocp-async-2,8661c0b564eaf2b4,30bdba19a304c1b6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:44.707 INFO 2458 --- [prometheus-minute-schedule-1,d708800dbafc41fe,a87f38f3f432e64e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:44.714 INFO 2458 --- [prometheus-minute-schedule-1,d708800dbafc41fe,a87f38f3f432e64e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:45.102 INFO 2458 --- [prometheus-second-schedule-1,3e6e186f28c06791,a2fe99a84f97f7b3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:27:45.166 INFO 2458 --- [ocp-schedules-1,b22e5f3e6e368eee,ca6a0ce7ed550dba] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:45.170 INFO 2458 --- [ocp-schedules-1,b22e5f3e6e368eee,ca6a0ce7ed550dba] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:45.436 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:45.437 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:46.611 WARN 2458 --- [ocp-async-2,a8e06a97c15abf64,059a5d83ecd318a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:46.611 INFO 2458 --- [ocp-async-10,2f57319034f4b379,11d7d9be7c661a5b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:46.810 INFO 2458 --- [ocp-schedules-1,dcf689f474758e1c,218dcd072eea9645] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:47.480 INFO 2458 --- [ocp-schedules-23,58ee43e192045560,e69696b1e4da3a0d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:27:47.488 INFO 2458 --- [ocp-schedules-23,58ee43e192045560,e69696b1e4da3a0d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:47.488 INFO 2458 --- [ocp-schedules-23,58ee43e192045560,e69696b1e4da3a0d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:27:47.624 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:27:47.664 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:47.668 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:47.707 INFO 2458 --- [prometheus-minute-schedule-1,70755a809025579f,c0c73c07e31ffc44] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:47.714 INFO 2458 --- [prometheus-minute-schedule-1,70755a809025579f,c0c73c07e31ffc44] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:47.732 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:47.735 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:47.791 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:47.794 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:47.825 INFO 2458 --- [ocp-schedules-12,ddfa59388f992ff2,afddf3511b91022c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:47.846 WARN 2458 --- [scanner-112192349-1,ddfa59388f992ff2,b74cd18dc9fa8a73] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:47.846 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:47.850 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:47.917 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:47.920 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:47.931 INFO 2458 --- [alarm-detect-18,ddfa59388f992ff2,bf118c651ae2a669] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535635 ~ 1761535657 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535650, value=43.104)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535650, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535650=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:27:47.971 INFO 2458 --- [ocp-schedules-18,036528218d3cb5c9,b7821cad1e82da8e] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:27:48.437 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:48.438 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:48.535 INFO 2458 --- [ocp-async-10,e93060e12d4d85c7,e158cba94e6bd778] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:48.611 WARN 2458 --- [ocp-async-6,5496e98e5b3e9f2f,ef19544b553f0c63] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:48.612 INFO 2458 --- [ocp-async-8,c7f6168be16940ca,81307e8c0cbafb51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:49.421 INFO 2458 --- [ocp-schedules-12,de4b135672bbb015,0651a3d3a3630cd1] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:27:49.421+08:00, clusters=[1, 2] 2025-10-27 11:27:49.436 INFO 2458 --- [ocp-schedules-12,de4b135672bbb015,0651a3d3a3630cd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:27:49.438 INFO 2458 --- [ocp-schedules-12,de4b135672bbb015,0651a3d3a3630cd1] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:27:50.611 WARN 2458 --- [ocp-async-4,dba0fbc7cd19184c,4daa271055feb163] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:50.612 INFO 2458 --- [ocp-async-1,131b64e17ec35c38,b0dc540afe868dba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:50.707 INFO 2458 --- [prometheus-minute-schedule-1,17b3fa1517bf1fef,3b169dd66245119a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:50.715 INFO 2458 --- [prometheus-minute-schedule-1,17b3fa1517bf1fef,3b169dd66245119a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:51.412 INFO 2458 --- [ocp-schedules-17,fabd1023b27d2836,5cf55b7c623601c3] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:51.412 INFO 2458 --- [ocp-schedules-4,00995d11f9358842,9aee771094c657f6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:51.413 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:27:51.413 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:27:51.413 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:27:51.413 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:27:51.413 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:27:51.414 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:27:51.414 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:27:51.414 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:27:51.415 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:27:51.415 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:27:51.430 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:51.431 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:51.431 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:51.432 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:51.432 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:51.432 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:51.434 INFO 2458 --- [ocp-schedules-30,d6a956af7f9cea22,68d28c88832ee8c4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:27:51.434 INFO 2458 --- [host-check-clockdiff1,d6a956af7f9cea22,48b959ddeab8613b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:27:51.434 INFO 2458 --- [host-check-clockdiff2,d6a956af7f9cea22,4f32176dde4630b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:27:51.434 INFO 2458 --- [host-check-clockdiff3,d6a956af7f9cea22,681730f0fb4510df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:27:51.435 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:51.435 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:51.437 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-clockdiff5,d6a956af7f9cea22,fa64b332fd0fc13e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-clockdiff4,d6a956af7f9cea22,2faed6fa7ccec87b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:27:51.438 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:51.438 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:51.439 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:51.439 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:51.448 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:51.449 INFO 2458 --- [ocp-schedules-1,2e647bc4510f096d,375a719dd8c81330] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:27:51.449 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:51.455 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:51.456 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:51.460 INFO 2458 --- [host-check-clockdiff2,d6a956af7f9cea22,4f32176dde4630b6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535671 -1 -1 , error= 2025-10-27 11:27:51.460 INFO 2458 --- [host-check-clockdiff4,d6a956af7f9cea22,2faed6fa7ccec87b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535671 0 0 , error= 2025-10-27 11:27:51.460 INFO 2458 --- [host-check-clockdiff5,d6a956af7f9cea22,fa64b332fd0fc13e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535671 0 0 , error= 2025-10-27 11:27:51.461 INFO 2458 --- [host-check-clockdiff1,d6a956af7f9cea22,48b959ddeab8613b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535671 0 0 , error= 2025-10-27 11:27:51.461 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:51.520 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:27:51.521 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:27:51.521 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:27:51.523 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:27:51.522667903+08:00 2025-10-27 11:27:51.523 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:27:51.523+08:00 2025-10-27 11:27:51.524 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:27:51.524 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:27:51.522793711+08:00 2025-10-27 11:27:51.524 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:27:51.525 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:27:51.524+08:00 2025-10-27 11:27:51.525 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:27:51.525 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:27:51.525 INFO 2458 --- [host-check-agent-status5,00995d11f9358842,6fae2b4f1cfbdd93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:27:51.526 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:27:51.524433389+08:00 2025-10-27 11:27:51.526 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:27:51.526+08:00 2025-10-27 11:27:51.528 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:27:51.528 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:27:51.528 INFO 2458 --- [host-check-agent-status2,00995d11f9358842,69d06d70e776fb18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:27:51.528 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:27:51.532 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:27:51.532 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:27:51.532 INFO 2458 --- [host-check-agent-status3,00995d11f9358842,7bb1ae624ebcbc05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:27:51.532 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:27:51.532 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:27:51.523+08:00 and status AVAILABLE 2025-10-27 11:27:51.534 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:27:51.524+08:00 and status AVAILABLE 2025-10-27 11:27:51.535 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:27:51.526+08:00 and status AVAILABLE 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:27:51.536 INFO 2458 --- [ocp-schedules-14,09d7084645336313,45854007a31115d8] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-agent-status4,00995d11f9358842,59f3a60f2289d153] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:27:51.536 INFO 2458 --- [ocp-schedules-14,09d7084645336313,45854007a31115d8] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:27:51.536 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:27:51.535916160+08:00 2025-10-27 11:27:51.537 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:27:51.536592629+08:00 2025-10-27 11:27:51.537 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:27:51.537 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:27:51.536+08:00 2025-10-27 11:27:51.538 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:27:51.537+08:00 2025-10-27 11:27:51.539 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:27:51.541 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:27:51.542 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:27:51.545 INFO 2458 --- [ocp-schedules-22,c01a0fceb1c0e927,ec1e2f1fd5e75835] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:27:51.546 INFO 2458 --- [ocp-schedules-14,09d7084645336313,45854007a31115d8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:27:51.546 INFO 2458 --- [ocp-schedules-14,09d7084645336313,45854007a31115d8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:27:51.546 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:27:51.537+08:00 and status AVAILABLE 2025-10-27 11:27:51.549 INFO 2458 --- [ocp-schedules-10,657301c2f2c36173,3e69b8adcbf4af11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:27:51.549 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:27:51.536+08:00 and status AVAILABLE 2025-10-27 11:27:51.550 INFO 2458 --- [ocp-schedules-22,c01a0fceb1c0e927,ec1e2f1fd5e75835] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:51.552 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:27:51.553 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:27:51.553 INFO 2458 --- [ocp-schedules-10,657301c2f2c36173,3e69b8adcbf4af11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:51.553 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:27:51.553 INFO 2458 --- [host-check-agent-status1,00995d11f9358842,8318302004d9140c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:27:51.554 INFO 2458 --- [ocp-schedules-2,38d8156d1c0e141a,7190159ca042be91] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:27:51.554 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:27:51.556 INFO 2458 --- [ocp-schedules-10,657301c2f2c36173,3e69b8adcbf4af11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:27:51.557 INFO 2458 --- [ocp-schedules-2,38d8156d1c0e141a,7190159ca042be91] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:51.565 INFO 2458 --- [ocp-schedules-29,4f741c926def8860,b501ff66ab83da50] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:27:51.566 INFO 2458 --- [ocp-schedules-3,0477c3509069f1da,c2e227e1795bafbc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:27:51.568 INFO 2458 --- [ocp-schedules-18,379e44f1a8ef6afe,a4aa44f345a6a5e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:27:51.569 INFO 2458 --- [ocp-schedules-5,539ebc7b5e068b1c,df42cb820114e97a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:27:51.574 INFO 2458 --- [ocp-schedules-3,0477c3509069f1da,c2e227e1795bafbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:51.577 INFO 2458 --- [ocp-schedules-18,379e44f1a8ef6afe,a4aa44f345a6a5e9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:27:51.578 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:27:51.580 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:51.581 INFO 2458 --- [ocp-schedules-5,539ebc7b5e068b1c,df42cb820114e97a] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:27:51.581 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:27:51.583 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:51.595 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:27:51.598 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:51.598 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:27:51.601 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:51.602 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:27:51.605 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:27:51.625 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.625 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:51.626 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:27:51.638 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.639 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:27:51.647 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,b351c4df52cb54b2,063f3632d2de55af] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=b351c4df52cb54b2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:27:51.647 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,b351c4df52cb54b2,063f3632d2de55af] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=b351c4df52cb54b2, duration=1 ms] 2025-10-27 11:27:51.649 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.650 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:27:51.651 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.651 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.653 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:51.657 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.658 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.658 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.658 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.658 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.658 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.659 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:27:51.660 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:27:51.662 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:27:51.663 INFO 2458 --- [ocp-schedules-22,c01a0fceb1c0e927,ec1e2f1fd5e75835] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:27:51.679 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:51.683 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:51.695 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:51.700 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,b80f454530f29c5f,f1f16f784153d582] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=b80f454530f29c5f, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:27:51.700 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,b80f454530f29c5f,f1f16f784153d582] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=b80f454530f29c5f, duration=1 ms] 2025-10-27 11:27:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.702 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.702 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.703 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:27:51.706 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:27:51.711 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:51.721 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:51.721 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:27:51.726 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:27:51.730 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:27:51.728522+08:00, lastProcessTime=2025-10-27T11:27:41.712686+08:00 2025-10-27 11:27:51.731 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:27:51.737 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:27:51.740 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.740 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.741 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.743 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:27:51.743 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:27:51.745 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.745 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.745 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.748 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:27:51.749 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.749 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.749 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.749 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.749 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.750 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:51.750 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.750 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.750 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.753 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:27:51.755 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:27:51.756 INFO 2458 --- [ocp-schedules-3,0477c3509069f1da,c2e227e1795bafbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:27:51.757 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:51.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:27:51.762 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:27:51.766 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:27:51.771 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:27:51.778 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:27:51.787 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:51.802 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:51.809 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:51.814 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:51.815 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:27:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.822 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:27:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:51.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.824 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:27:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:27:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:51.829 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:27:51.832 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:51.832 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.832 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.832 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.832 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.833 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.833 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:27:51.835 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:27:51.842 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:27:51.853 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:27:51.855 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:27:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:51.859 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:51.864 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:27:51.872 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:27:51.875 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:27:51.879 INFO 2458 --- [ocp-schedules-11,11a4e17f57f77aac,e246b1a3c465974c] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:27:51.879 INFO 2458 --- [ocp-schedules-11,11a4e17f57f77aac,e246b1a3c465974c] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:27:51.885 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:51.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:51.892 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:51.901 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:51.906 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:51.912 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:27:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:51.917 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:27:51.917 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.918 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:51.918 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:27:51.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:27:51.920 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:27:51.922 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:27:51.931 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:27:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.936 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:27:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:51.938 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:27:51.940 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:27:51.942 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:27:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:51.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:27:51.952 INFO 2458 --- [host-check-host-status4,fabd1023b27d2836,eb834285da1d2a5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:27:51.952 INFO 2458 --- [host-check-host-status3,fabd1023b27d2836,cff1b01edfe0cbbb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:27:51.954 INFO 2458 --- [host-check-host-status1,fabd1023b27d2836,c7f9055e4170ceb1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:27:51.955 INFO 2458 --- [host-check-host-status5,fabd1023b27d2836,7b00382cfaeb2727] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:27:51.967 INFO 2458 --- [host-check-host-status2,fabd1023b27d2836,3ee96d1169bb9810] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:27:51.982 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:51.982 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:51.991 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:27:51.728522+08:00 2025-10-27 11:27:52.005 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:27:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:52.049 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:27:52.049 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:27:52.051 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:27:52.056 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:52.056 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:52.068 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:27:52.092 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:52.092 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.093 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:27:52.094 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:52.094 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:27:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:27:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:27:52.121 INFO 2458 --- [pool-overview-container-7,b74853dc26ccaa14,7de2e0afeef8faa2] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:52.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:27:52.168 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.209 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.210 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:27:52.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:27:52.222 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:27:52.233 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:27:52.235 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:27:52.238 INFO 2458 --- [pool-overview-container-2,820794600a05328f,67dfa6c2983a1cfa] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.278 INFO 2458 --- [pool-overview-container-4,6845089c79297a1e,e0b0f633293c0015] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.280 INFO 2458 --- [pool-overview-container-4,6845089c79297a1e,e0b0f633293c0015] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:27:52.295 INFO 2458 --- [pool-12-thread-2,820794600a05328f,beb04f00b1b66507] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:27:52.297 INFO 2458 --- [pool-12-thread-1,820794600a05328f,a6c8cf3e735c9683] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:27:52.298 INFO 2458 --- [pool-12-thread-4,820794600a05328f,fdaf6c145eb5315b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:27:52.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:27:52.303 INFO 2458 --- [pool-12-thread-3,820794600a05328f,b3c0e114d5df546b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:27:52.303 INFO 2458 --- [pool-12-thread-2,820794600a05328f,beb04f00b1b66507] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:27:52.303 INFO 2458 --- [pool-12-thread-2,820794600a05328f,beb04f00b1b66507] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:27:52.304 INFO 2458 --- [ForkJoinPool-7-worker-6,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.305 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:27:52.305 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:27:52.305 INFO 2458 --- [ForkJoinPool-7-worker-15,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.305 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:27:52.305 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:27:52.321 INFO 2458 --- [pool-12-thread-4,820794600a05328f,fdaf6c145eb5315b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:27:52.328 INFO 2458 --- [ForkJoinPool-7-worker-8,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.328 INFO 2458 --- [ForkJoinPool-7-worker-1,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:27:52.348 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:27:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.349 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.349 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:27:52.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.351 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.352 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:27:52.355 INFO 2458 --- [pool-12-thread-4,820794600a05328f,fdaf6c145eb5315b] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:27:52.360 INFO 2458 --- [pool-12-thread-4,820794600a05328f,fdaf6c145eb5315b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:27:52.365 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.366 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:27:52.366 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.367 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.367 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:27:52.369 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:27:52.377 INFO 2458 --- [host-check-clockdiff3,d6a956af7f9cea22,681730f0fb4510df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535672 0 0 , error= 2025-10-27 11:27:52.385 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.385 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.385 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.385 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.388 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.388 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:27:52.389 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:27:52.392 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.392 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.393 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.394 INFO 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:27:52.396 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:27:52.405 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:27:52.405 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.407 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:27:52.408 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.409 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:27:52.417 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.417 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:27:52.418 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.419 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:27:52.419 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:27:52.446 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.446 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.447 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.447 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.447 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.448 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:27:52.451 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:27:52.452 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:27:52.452 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:27:52.452 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.453 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.453 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.453 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.453 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.454 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:27:52.455 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.455 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.456 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.459 WARN 2458 --- [ForkJoinPool-7-worker-15,820794600a05328f,63dbc9ceaf000ee2] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:27:52.459 INFO 2458 --- [ForkJoinPool-7-worker-1,820794600a05328f,6ad2b17be0b7a1dc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:27:52.459 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.460 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:27:52, 2025-10-27 11:27:52] 2025-10-27 11:27:52.462 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.464 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:27:52.489 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:27:52.490 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.491 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:27:52, 2025-10-27 11:27:52] 2025-10-27 11:27:52.493 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:27:52.493 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.493 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:27:52.493 INFO 2458 --- [ForkJoinPool-7-worker-6,820794600a05328f,64a62d049ffe019b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:27:52.493 INFO 2458 --- [pool-12-thread-1,820794600a05328f,a6c8cf3e735c9683] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:27:52.495 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:27:52.495 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.496 INFO 2458 --- [ForkJoinPool-7-worker-8,820794600a05328f,faca200991cc44db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:27:52.500 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:27:52.500 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:27:52.501 INFO 2458 --- [ocp-schedules-2,38d8156d1c0e141a,7190159ca042be91] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:27:52.511 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.511 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.511 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.511 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.512 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.516 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:27:52.519 INFO 2458 --- [pool-12-thread-3,820794600a05328f,b3c0e114d5df546b] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:27:52.527 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:52.532 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.537 INFO 2458 --- [refresh-tenant-resource-cache-7,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:27:52.611 WARN 2458 --- [ocp-async-10,56ee14210a8940a1,c92021b696613594] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:52.612 INFO 2458 --- [ocp-async-6,03738b103d9eac81,d94f420852949c1a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:52.667 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:27:52.668 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.668 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.668 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:27:52.668 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:27:52.668 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:27:52.673 INFO 2458 --- [refresh-tenant-resource-cache-5,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:27:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:27:53.235 INFO 2458 --- [ocp-schedules-11,89506e19c223e94c,37ee2680d75d13b7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20387,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:27:53.235 INFO 2458 --- [ocp-schedules-11,89506e19c223e94c,37ee2680d75d13b7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:27:53.236 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.248 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterVi 2025-10-27 11:27:53.250 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.252 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.253 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.254 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.255 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.256 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.358 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.359 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.360 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.493 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.498 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.707 INFO 2458 --- [prometheus-minute-schedule-1,ef8e6bf412193792,aecc4d0fe7f5e1cb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:53.724 INFO 2458 --- [prometheus-minute-schedule-1,ef8e6bf412193792,aecc4d0fe7f5e1cb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:53.894 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.897 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.900 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.902 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.904 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.905 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.906 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.907 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.908 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.915 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.916 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:27:53.919 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:53.924 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 7 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.113 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.114 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.115 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:27:54.116 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.117 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.118 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:21:52.919, end=2025-10-27T11:26:53.229} 2025-10-27 11:27:54.118 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 1 dubious sqls 2025-10-27 11:27:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:27:54.439 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:54.440 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:54.612 WARN 2458 --- [ocp-async-3,9efa52b00b2b3e54,5f3e3d1b8a6dd996] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:54.612 INFO 2458 --- [ocp-async-4,1f9a207600881874,9c31031279053a4f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:55.177 INFO 2458 --- [ocp-schedules-12,94fc5723355b98b0,fb271495fb9d7b34] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:27:55.181 INFO 2458 --- [ocp-schedules-12,94fc5723355b98b0,fb271495fb9d7b34] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:27:55.375 INFO 2458 --- [ocp-schedules-18,701a3355f6ee42d4,0316b8aa15c18412] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:27:56.612 WARN 2458 --- [ocp-async-8,ef1abe0349398ceb,4785bba4a0591cde] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:56.612 INFO 2458 --- [ocp-async-9,cb66536c919c43d7,c473ef01feec0714] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:56.707 INFO 2458 --- [prometheus-minute-schedule-1,7f13cbe30a41f4d7,7799545a4ed1510d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:56.718 INFO 2458 --- [prometheus-minute-schedule-1,7f13cbe30a41f4d7,7799545a4ed1510d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:27:56.818 INFO 2458 --- [ocp-schedules-5,d299e9ff1884c742,071d74c15be85f6a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:27:57.440 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:27:57.441 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:27:57.942 INFO 2458 --- [ocp-schedules-4,394c392d712d0f1d,1ece408712b8ad7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:27:57.989 WARN 2458 --- [scanner-112192550-1,394c392d712d0f1d,f7fdf3e0a16e095a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:27:58.047 INFO 2458 --- [alarm-detect-2,394c392d712d0f1d,d95eab6efb55b91f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535608 ~ 1761535668 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535620, value=203954.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=203954.56, values={1761535620=203954.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:27:58.103 INFO 2458 --- [alarm-detect-2,394c392d712d0f1d,676ce7bbbecdd849] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535608 ~ 1761535668 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535620, value=93.159)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.159, values={1761535620=93.159}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:27:58.132 INFO 2458 --- [alarm-detect-19,394c392d712d0f1d,149a4f93ad6f6844] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535608 ~ 1761535668 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535620, value=203955.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=203955.912, values={1761535620=203955.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:27:58.138 INFO 2458 --- [alarm-detect-15,394c392d712d0f1d,9ccc46d53c5c6e32] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535650 ~ 1761535668 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535665, value=46.516)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535665, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535665=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:27:58.240 INFO 2458 --- [ocp-schedules-9,84d73acf50959bc5,800b55afca43bf4d] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:27:58.248 INFO 2458 --- [ocp-schedules-9,84d73acf50959bc5,800b55afca43bf4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:27:58.294 INFO 2458 --- [ocp-schedules-9,84d73acf50959bc5,800b55afca43bf4d] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:27:58.535 INFO 2458 --- [ocp-async-1,c7b252024e0eb626,e113c5717028ae16] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:27:58.612 WARN 2458 --- [ocp-async-5,91a750ae3e78503e,3568054527732bba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:27:58.612 INFO 2458 --- [ocp-async-7,b727f67eac6a1e22,3f73714f4d000ce8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:27:59.456 INFO 2458 --- [ocp-schedules-4,f6af5d24e98ad7e1,1ba3806b9afd098e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:27:59.614 INFO 2458 --- [ocp-schedules-22,d50e6d29ac691a7b,ef459ee94ff4c3a0] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:27:59.615 INFO 2458 --- [ocp-schedules-22,d50e6d29ac691a7b,ef459ee94ff4c3a0] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:27:59.707 INFO 2458 --- [prometheus-minute-schedule-1,a3f3f4f4e841028a,1bdc5c9e12b16cd7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535680, interval=60 2025-10-27 11:27:59.716 INFO 2458 --- [prometheus-minute-schedule-1,a3f3f4f4e841028a,1bdc5c9e12b16cd7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:00.441 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:00.442 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:00.612 WARN 2458 --- [ocp-async-6,d721965cec123f46,8eedfac84b250911] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:00.612 INFO 2458 --- [ocp-async-2,373b816af085db11,b3bcef0f09fb446a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:01.024 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,cb63c07ecf8137db,993850b07b22ac5d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=cb63c07ecf8137db, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:28:01.024 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,cb63c07ecf8137db,993850b07b22ac5d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=cb63c07ecf8137db, duration=1 ms] 2025-10-27 11:28:01.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:01.717728+08:00, lastProcessTime=2025-10-27T11:27:51.728522+08:00 2025-10-27 11:28:01.941 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:01.717728+08:00 2025-10-27 11:28:02.612 WARN 2458 --- [ocp-async-4,a70fd1464d667d20,eabd143032122b6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:02.612 INFO 2458 --- [ocp-async-9,88a575153a04c417,321f17a897264d3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:02.707 INFO 2458 --- [prometheus-minute-schedule-1,d1eecae4cb3d3954,4169fe09efe861ae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:02.717 INFO 2458 --- [prometheus-minute-schedule-1,d1eecae4cb3d3954,4169fe09efe861ae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:03.442 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:03.443 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:04.130 INFO 2458 --- [ocp-schedules-26,a9fc168cb97484d8,381270601fc34263] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:28:04.139 INFO 2458 --- [ocp-schedules-26,a9fc168cb97484d8,381270601fc34263] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:04.179 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:04.180 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:28:04.193 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:04.193 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:04.193 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:04.193 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:04.194 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:04.194 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:28:04.207 INFO 2458 --- [ocp-schedules-26,a9fc168cb97484d8,381270601fc34263] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:28:04.612 WARN 2458 --- [ocp-async-10,28b503271e045214,022d3176870fdb11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:04.613 INFO 2458 --- [ocp-async-3,8c9c834817c9d201,86e8e21130abbdf0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:05.190 INFO 2458 --- [ocp-schedules-4,d0c7804b00c42ee6,82e34bbda2258cad] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:05.195 INFO 2458 --- [ocp-schedules-4,d0c7804b00c42ee6,82e34bbda2258cad] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:05.707 INFO 2458 --- [prometheus-minute-schedule-1,3833df707fb2b9b0,ed2f2a2320ca8a97] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:05.715 INFO 2458 --- [prometheus-minute-schedule-1,3833df707fb2b9b0,ed2f2a2320ca8a97] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:28:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:28:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:28:06.069 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:28:06.112 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:28:06.135 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:28:06.135 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:28:06.135 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:28:06.135 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:28:06.139 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:28:06.140 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:06.143 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:28:06.145 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:06.146 INFO 2458 --- [sql-rollup-worker-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:28:06.384 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:25:30, start=2025-10-27T11:22:30, end=2025-10-27T11:25:00, updates=330, granule=120000ms, delay=180000ms 2025-10-27 11:28:06.397 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:25:30, target=2025-10-27T11:25:00, granule=120000ms, delay=30000ms) 2025-10-27 11:28:06.397 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:25:00, target=2025-10-27T11:12:30, granule=600000ms, delay=750000ms) 2025-10-27 11:28:06.397 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=393ms, period=120000ms 2025-10-27 11:28:06.443 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:06.444 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:06.612 WARN 2458 --- [ocp-async-6,bf1ff16bccacf2af,c31f3777db27ce5e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:06.613 INFO 2458 --- [ocp-async-2,93ce4622339be09a,07338f27a2a17eb3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:06.825 INFO 2458 --- [ocp-schedules-4,8cc803ffb25137c8,9c5bc1dca5ca0128] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:07.490 INFO 2458 --- [ocp-schedules-12,9e0e4262c8ebbdfe,dbb6d6792567e12d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:28:07.498 INFO 2458 --- [ocp-schedules-12,9e0e4262c8ebbdfe,dbb6d6792567e12d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:07.498 INFO 2458 --- [ocp-schedules-12,9e0e4262c8ebbdfe,dbb6d6792567e12d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:08.145 INFO 2458 --- [ocp-schedules-29,9a55e08d847080dd,f6701963b797d849] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:08.174 WARN 2458 --- [scanner-112192790-1,9a55e08d847080dd,c4929f7d5ba6d81d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:08.535 INFO 2458 --- [ocp-async-2,695c6093bcaaa518,8e490c2b27a4c360] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:08.613 WARN 2458 --- [ocp-async-8,e497894a97c92c31,1c8391bde8acba9c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:08.613 INFO 2458 --- [ocp-async-1,427aae4d6bafeb07,3625a046e18277b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:08.707 INFO 2458 --- [prometheus-minute-schedule-1,533950d42462591c,9cd6b618dd4cc368] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:08.716 INFO 2458 --- [prometheus-minute-schedule-1,533950d42462591c,9cd6b618dd4cc368] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:09.444 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:09.445 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:10.015 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:28:10.017 INFO 2458 --- [ocp-schedules-18,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:28:10.061 INFO 2458 --- [Collect obproxy1,108547d206dbca8b,13b18cdd1a795e72] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:28:10.061 INFO 2458 --- [Check obproxy1,6aba6c70b7c9bc3c,0585c53c4e97c362] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:28:10.061 INFO 2458 --- [Check obproxy1,6aba6c70b7c9bc3c,0585c53c4e97c362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:28:10.061 INFO 2458 --- [Collect obproxy1,108547d206dbca8b,13b18cdd1a795e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:28:10.092 INFO 2458 --- [Collect obproxy1,108547d206dbca8b,13b18cdd1a795e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:28:10.093 INFO 2458 --- [Check obproxy1,6aba6c70b7c9bc3c,0585c53c4e97c362] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:28:10.096 INFO 2458 --- [Collect obproxy1,108547d206dbca8b,13b18cdd1a795e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:28:10.097 INFO 2458 --- [Check obproxy1,6aba6c70b7c9bc3c,0585c53c4e97c362] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:28:10.097 INFO 2458 --- [Check obproxy1,6aba6c70b7c9bc3c,0585c53c4e97c362] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:28:10.099 INFO 2458 --- [Collect obproxy1,108547d206dbca8b,13b18cdd1a795e72] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:28:10.179 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:28:10.300 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=1000, affectRows=1000 2025-10-27 11:28:10.304 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=994, validCount=992 2025-10-27 11:28:10.390 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=1000, affectRows=1000 2025-10-27 11:28:10.394 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=999, validCount=995 2025-10-27 11:28:10.407 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587947) Duplicate entry '835370602979598336-1761535560' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587947) Duplicate entry '835370602979598336-1761535560' for key 'PRIMARY' 2025-10-27 11:28:10.618 WARN 2458 --- [ocp-async-10,80cee0fb8c3ef325,047b4ca7a490036d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:10.619 INFO 2458 --- [ocp-async-5,fc5e3a340b9c3540,8fac704283aa7979] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:10.687 INFO 2458 --- [ocp-schedules-4,91c21ebdf9650532,74faca1ea348d566] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:28:10.696 INFO 2458 --- [ocp-schedules-4,91c21ebdf9650532,74faca1ea348d566] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:10.794 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:28:10.794 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.800 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:28:10.810 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.811 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.841 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.842 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.872 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.874 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.875 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:10.877 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:28:10.894 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.894 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.895 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.896 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.915 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.916 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:10.931 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.931 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.931 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.932 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.954 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.955 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:10.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.969 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.969 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.970 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:28:10.972 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:10.972 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:10.972 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.973 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:10.990 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.991 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:10.992 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.993 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:10.993 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.994 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:10.995 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.996 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:10.996 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.997 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:10.998 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:10.999 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:10.999 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:28:11.002 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:28:11.005 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.005 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.006 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.006 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:11.037 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.037 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.037 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.038 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:28:11.040 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.040 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.040 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.041 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:28:11.066 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:11.067 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.068 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:28:11.076 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.076 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.076 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.077 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:28:11.078 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.078 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:28:11.078 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.078 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:28:11.081 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:28:11.081 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:28:11.118 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.118 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.118 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.119 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:28:11.122 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.122 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.122 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.123 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:28:11.166 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.166 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.166 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.167 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:28:11.196 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.196 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.198 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:28:11.250 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.250 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:11.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.252 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.253 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:11.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.255 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.256 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:11.257 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.257 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.257 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:11.259 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.259 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:11.260 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.260 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:11.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.262 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:11.263 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.263 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.264 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.265 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:11.265 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.265 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:11.267 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.267 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.268 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:28:11.268 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.268 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.269 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.270 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:28:11.270 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.270 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.270 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.271 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:28:11.272 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:28:11.275 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:28:11.376 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:11.376 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.376 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:11.377 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:11.377 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:11.377 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:28:11.405 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:28:11.409 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:28:11.409 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:28:11.410 INFO 2458 --- [ocp-schedules-4,91c21ebdf9650532,74faca1ea348d566] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:28:11.707 INFO 2458 --- [prometheus-minute-schedule-1,d81d8ed5ed457252,0ce8abf37f8f904b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:11.714 INFO 2458 --- [prometheus-minute-schedule-1,d81d8ed5ed457252,0ce8abf37f8f904b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:11.714284+08:00, lastProcessTime=2025-10-27T11:28:01.717728+08:00 2025-10-27 11:28:11.859 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535620, nodeCount=817, affectRows=817 2025-10-27 11:28:11.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:11.714284+08:00 2025-10-27 11:28:12.445 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:12.446 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:12.618 WARN 2458 --- [ocp-async-6,994cf3606c1dc127,d3fd910d084c2931] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:12.619 INFO 2458 --- [ocp-async-7,5aa7ca129886c862,57a2a3f1787acc1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:12.972 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=1000, affectRows=1000 2025-10-27 11:28:12.976 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=1000, validCount=996 2025-10-27 11:28:12.983 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=546, affectRows=546 2025-10-27 11:28:12.985 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=544, validCount=542 2025-10-27 11:28:12.993 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=551, affectRows=551 2025-10-27 11:28:12.995 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=549, validCount=547 2025-10-27 11:28:14.489 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535620, nodeCount=1000, affectRows=1000 2025-10-27 11:28:14.493 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535620, nodeCount=995, validCount=982 2025-10-27 11:28:14.619 WARN 2458 --- [ocp-async-2,eeefcffb09c2970d,889c8655a965177a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:14.619 INFO 2458 --- [ocp-async-2,28dfd57f86221b97,4082660428010562] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:14.707 INFO 2458 --- [prometheus-minute-schedule-1,ed4d9ba4f99127c6,f366332b00eb2101] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:14.715 INFO 2458 --- [prometheus-minute-schedule-1,ed4d9ba4f99127c6,f366332b00eb2101] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:15.203 INFO 2458 --- [ocp-schedules-10,74dc027fc5ad7a98,68ffc663975369a4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:15.208 INFO 2458 --- [ocp-schedules-10,74dc027fc5ad7a98,68ffc663975369a4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:15.446 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:15.447 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:15.782 INFO 2458 --- [ocp-schedules-25,2f8a5c810abf697d,11ae7c94fcdef18c] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:28:15.782+08:00, clusters=[1, 2] 2025-10-27 11:28:15.786 INFO 2458 --- [ocp-schedules-25,2f8a5c810abf697d,11ae7c94fcdef18c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:28:15.788 INFO 2458 --- [ocp-schedules-25,2f8a5c810abf697d,11ae7c94fcdef18c] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:28:16.619 WARN 2458 --- [ocp-async-1,ff1e738559f7a1a7,83dbe6c25cac9cca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:16.619 INFO 2458 --- [ocp-async-1,e235421fef3c8bb7,09894fc695345177] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:16.836 INFO 2458 --- [ocp-schedules-30,6b6659a5c7ed17ba,0ce3c73ccbc07cd5] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:17.440 INFO 2458 --- [ocp-schedules-27,ef91db63e99d9f2a,c326005594801dd1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:28:17.440 INFO 2458 --- [ocp-schedules-27,ef91db63e99d9f2a,c326005594801dd1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:28:17.500 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:28:17.500 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:28:17.500 INFO 2458 --- [Collect all tenants major compactions5,ef91db63e99d9f2a,20695f82ec9fc5b7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:28:17.500 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:28:17.500 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:28:17.501 INFO 2458 --- [Collect all tenants major compactions6,ef91db63e99d9f2a,5c4417f7320faf34] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:28:17.538 INFO 2458 --- [Collect all tenants major compactions5,ef91db63e99d9f2a,20695f82ec9fc5b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.538 INFO 2458 --- [Collect all tenants major compactions6,ef91db63e99d9f2a,5c4417f7320faf34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.540 INFO 2458 --- [Collect all tenants major compactions6,ef91db63e99d9f2a,5c4417f7320faf34] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:28:17.540 INFO 2458 --- [Collect all tenants major compactions5,ef91db63e99d9f2a,20695f82ec9fc5b7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:28:17.543 INFO 2458 --- [Collect all tenants major compactions6,ef91db63e99d9f2a,5c4417f7320faf34] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:28:17.543 INFO 2458 --- [Collect all tenants major compactions5,ef91db63e99d9f2a,20695f82ec9fc5b7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:28:17.557 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.557 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.557 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.557 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.557 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.558 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.559 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.559 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:17.560 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:17.561 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:28:17.563 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:28:17.563 INFO 2458 --- [Collect all tenants major compactions2,ef91db63e99d9f2a,6aba4f121bf7d678] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:28:17.563 INFO 2458 --- [Collect all tenants major compactions3,ef91db63e99d9f2a,a2129a52a695fd26] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:28:17.565 INFO 2458 --- [Collect all tenants major compactions4,ef91db63e99d9f2a,36393f6252df7195] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:28:17.566 INFO 2458 --- [Collect all tenants major compactions1,ef91db63e99d9f2a,0ebb5914ed3f9919] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:28:17.567 INFO 2458 --- [ocp-schedules-27,ef91db63e99d9f2a,c326005594801dd1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:28:17.707 INFO 2458 --- [prometheus-minute-schedule-1,9ae8d436e856280f,69583b0a73a28f8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:17.715 INFO 2458 --- [prometheus-minute-schedule-1,9ae8d436e856280f,69583b0a73a28f8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:17.894 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b67c1c3b6c26ecf2,a95e38c845e28f5a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b67c1c3b6c26ecf2, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:28:17.894 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b67c1c3b6c26ecf2,a95e38c845e28f5a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b67c1c3b6c26ecf2, duration=1 ms] 2025-10-27 11:28:18.290 INFO 2458 --- [ocp-schedules-24,ac6f4adc6b3f571f,88c92f473143721f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:18.358 WARN 2458 --- [scanner-112194537-1,ac6f4adc6b3f571f,b22ccf7c2bedb7cb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:18.447 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:18.447 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:18.522 INFO 2458 --- [alarm-detect-9,ac6f4adc6b3f571f,193a1d1e0b73c6d4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535665 ~ 1761535688 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535680, value=74.412)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535680, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535680=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:28:18.535 INFO 2458 --- [ocp-async-1,28294a64cc969931,62613acf3830c9d9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:18.619 INFO 2458 --- [ocp-async-6,f1f78dfe96de68e1,a596b7b8a80c2f39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:18.619 WARN 2458 --- [ocp-async-7,2a6a97e6c7ebfd61,0177dffd32b2ecbd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:20.366 INFO 2458 --- [ocp-schedules-13,788cc2d88a14efec,74fbf8a0a354b1f2] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:28:20.619 INFO 2458 --- [ocp-async-9,23af9848271dd952,28f49cc4a48db7b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:20.619 WARN 2458 --- [ocp-async-2,6b8c670181c2dc0d,42010a1509fe73bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:20.707 INFO 2458 --- [prometheus-minute-schedule-1,ec8824afb8ae6c16,f9bb4075779aad09] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:20.715 INFO 2458 --- [prometheus-minute-schedule-1,ec8824afb8ae6c16,f9bb4075779aad09] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:20.969 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964} 2025-10-27 11:28:20.977 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo= 2025-10-27 11:28:20.977 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:20.980 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:20.980 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:28:20.980 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:20.984 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:20.985 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:20.985 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:28:20.985 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:20.988 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:20.989 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:20.989 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:28:20.989 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:20.992 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:20.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:20.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:28:20.998 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:21.026 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:21.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:21.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:28:21.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:21.154 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449301154000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:28:21.154Z, endTime=2025-10-27T03:26:20.964Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:28:21.157 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449301157000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:28:21.157Z, endTime=2025-10-27T03:26:20.964Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:28:21.171 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:28:21.171 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:28:21.199 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:28:21.199 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:28:21.269 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:28:21.269 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:28:21.296 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:28:21.296 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:28:21.300 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:28:21.300 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:28:21.154, end=2025-10-27T11:26:20.964}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:28:21.408 INFO 2458 --- [ocp-schedules-25,6418e83e252428d4,49932bb6fd1fccbe] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:21.408 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:28:21.409 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:28:21.409 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:28:21.409 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:28:21.409 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:28:21.410 INFO 2458 --- [ocp-schedules-30,e2732544f0277dfd,ef3af37b891209a4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:21.410 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:28:21.410 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:28:21.411 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:28:21.413 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:28:21.416 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:28:21.424 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:21.425 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:21.426 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:21.426 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:21.426 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:21.426 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:21.426 INFO 2458 --- [ocp-schedules-3,321dab46146b1326,258e9c8c52606347] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:21.427 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:21.427 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:21.427 INFO 2458 --- [host-check-clockdiff1,321dab46146b1326,5401f0e50be86421] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:28:21.427 INFO 2458 --- [host-check-clockdiff2,321dab46146b1326,fbb556961880c697] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:28:21.427 INFO 2458 --- [host-check-clockdiff5,321dab46146b1326,4fff803dca6fec66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:28:21.428 INFO 2458 --- [host-check-clockdiff4,321dab46146b1326,f8611c01a40db259] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:28:21.428 INFO 2458 --- [host-check-clockdiff3,321dab46146b1326,c62f1437d39907cb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:28:21.428 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:21.429 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:21.429 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:21.431 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:21.431 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:21.431 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:21.431 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:21.431 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:21.435 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:21.437 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:21.446 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:21.448 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:21.448 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:21.448 INFO 2458 --- [host-check-clockdiff3,321dab46146b1326,c62f1437d39907cb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535701 -1 -1 , error= 2025-10-27 11:28:21.443 INFO 2458 --- [ocp-schedules-11,1327391ca73e2ad5,09891acdfcecda2d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:28:21.449 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:21.449 INFO 2458 --- [host-check-clockdiff4,321dab46146b1326,f8611c01a40db259] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535701 -1 -1 , error= 2025-10-27 11:28:21.454 INFO 2458 --- [host-check-clockdiff2,321dab46146b1326,fbb556961880c697] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535701 -1 -1 , error= 2025-10-27 11:28:21.458 INFO 2458 --- [host-check-clockdiff1,321dab46146b1326,5401f0e50be86421] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535701 0 0 , error= 2025-10-27 11:28:21.466 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2025-10-27 11:28:21.466 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:28:21.495 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:28:21.495 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:28:21.503 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:28:21.503 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:28:21.506 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:28:21.506 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:28:21.508 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:28:21.507454968+08:00 2025-10-27 11:28:21.508 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:28:21.509 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:28:21.508+08:00 2025-10-27 11:28:21.509 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:28:21.510 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:28:21.510213844+08:00 2025-10-27 11:28:21.511 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:28:21.510+08:00 2025-10-27 11:28:21.511 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:28:21.511 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:28:21.511 INFO 2458 --- [host-check-agent-status3,6418e83e252428d4,c1745cc392e2231e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:28:21.514 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:28:21.514 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:28:21.514 INFO 2458 --- [host-check-agent-status2,6418e83e252428d4,4771782367a8a2ba] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:28:21.514 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:28:21.514 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:28:21.515 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:28:21.516 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:28:21.515542942+08:00 2025-10-27 11:28:21.516 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:28:21.516 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:28:21.516+08:00 2025-10-27 11:28:21.517 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:28:21.517 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:28:21.508+08:00 and status AVAILABLE 2025-10-27 11:28:21.518 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:28:21.518 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:28:21.518 INFO 2458 --- [host-check-agent-status5,6418e83e252428d4,fa28c1a3861fe337] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:28:21.518 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:28:21.518 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:28:21.517439270+08:00 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status4,6418e83e252428d4,d4f2685bba60c9a3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:28:21.519145223+08:00 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:28:21.518+08:00 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-agent-status1,6418e83e252428d4,630058366b9fabe3] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:28:21.519 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:28:21.519+08:00 2025-10-27 11:28:21.522 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:28:21.510+08:00 and status AVAILABLE 2025-10-27 11:28:21.523 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:28:21.525 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:28:21.516+08:00 and status AVAILABLE 2025-10-27 11:28:21.526 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:28:21.518+08:00 and status AVAILABLE 2025-10-27 11:28:21.526 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:28:21.527 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:28:21.519+08:00 and status AVAILABLE 2025-10-27 11:28:21.531 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:28:21.534 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:28:21.534 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:28:21.534 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535580964000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:28:21.535 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:28:21.539 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:28:21.545 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:21.545 INFO 2458 --- [ocp-schedules-23,e111ba4cc13e0359,cd9a1209cbef9706] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:28:21.546 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:21.548 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:21.549 INFO 2458 --- [ocp-schedules-15,3b0a4772f1648eab,1f022d491c1addfd] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:28:21.549 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:21.549 INFO 2458 --- [ocp-schedules-23,e111ba4cc13e0359,cd9a1209cbef9706] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:21.551 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:21.553 INFO 2458 --- [ocp-schedules-15,3b0a4772f1648eab,1f022d491c1addfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:21.553 INFO 2458 --- [ocp-schedules-15,3b0a4772f1648eab,1f022d491c1addfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:28:21.554 INFO 2458 --- [ocp-schedules-5,88c16831325eb910,e9f108dfc1183bcc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:28:21.554 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:21.554 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:21.556 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:21.556 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:21.557 INFO 2458 --- [ocp-schedules-5,88c16831325eb910,e9f108dfc1183bcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:21.559 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:21.566 INFO 2458 --- [ocp-schedules-13,2653a4d0a3a05df8,2acd7ef138581112] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:28:21.586 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:28:21.157, end=2025-10-27T11:26:20.964}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:28:21.587 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:28:21.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:24:20.235, end=2025-10-27T11:26:20.964},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:28:21.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:28:21.589 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:28:21.592 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:28:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:21.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:28:21.609 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:21.612 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:21.616 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:21.619 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.619 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.620 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.622 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:21.626 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:28:21.626 INFO 2458 --- [ocp-schedules-23,e111ba4cc13e0359,cd9a1209cbef9706] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:28:21.626 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:21.630 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:28:21.632 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:28:21.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.634 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:21.636 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:28:21.643 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:28:21.651 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:28:21.677 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:28:21.677 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:28:21.688 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:28:21.694 INFO 2458 --- [ocp-schedules-26,9a158436bb744af4,c28496cb8dc9752c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=1 2025-10-27 11:28:21.694 INFO 2458 --- [ocp-schedules-26,9a158436bb744af4,c28496cb8dc9752c] c.o.o.b.i.BinlogClusterSyncService : [OCP810] sync binlog cluster begin, binlogClusterId=4 2025-10-27 11:28:21.695 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:28:21.713 INFO 2458 --- [ocp-schedules-26,9a158436bb744af4,c28496cb8dc9752c] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:28:21.713 INFO 2458 --- [ocp-schedules-26,9a158436bb744af4,c28496cb8dc9752c] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:28:21.714 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:21.715 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.716 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:28:21.718 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:28:21.718 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.718 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.719 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.719 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.719 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.719 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:21.727 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:21.725933+08:00, lastProcessTime=2025-10-27T11:28:11.714284+08:00 2025-10-27 11:28:21.744 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:21.746 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:21.753 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:21.753 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:21.760 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:21.767 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:28:21.768 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:28:21.771 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:28:21.773 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:28:21.781 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:28:21.807 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:28:21.807 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:28:21.807 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:28:21.810 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:21.810 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.811 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:21.812 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.813 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:28:21.815 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:28:21.816 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:28:21.816 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:21.826 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:21.826 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.826 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.826 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.827 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.827 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.828 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:28:21.847 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:21.848 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:21.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:21.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:21.850 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:21.851 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:21.853 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:21.862 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:28:21.868 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:28:21.870 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:28:21.872 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:28:21.882 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:21.886 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:28:21.898 INFO 2458 --- [host-check-host-status2,e2732544f0277dfd,4b1130dfabccade6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:28:21.898 INFO 2458 --- [host-check-host-status5,e2732544f0277dfd,ad96b6b9903cc155] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:28:21.899 INFO 2458 --- [host-check-host-status4,e2732544f0277dfd,d3295e847b1772c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:28:21.907 INFO 2458 --- [host-check-host-status3,e2732544f0277dfd,f70cdd04e4ee7dd6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:28:21.909 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:21.909 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.911 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:28:21.919 INFO 2458 --- [host-check-host-status1,e2732544f0277dfd,72bee97f90882a45] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:28:21.924 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:28:21.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.944 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.944 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.944 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.945 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:21.946 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:21.949 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:28:21.951 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:22.007 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:28:22.013 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:21.725933+08:00 2025-10-27 11:28:22.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:22.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:22.051 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:28:22.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:28:22.054 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.055 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:28:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:22.071 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:28:22.094 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:22.095 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.096 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:28:22.104 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:22.104 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:22.118 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:28:22.118 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:28:22.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:22.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:22.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.220 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.221 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:22.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:28:22.234 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:28:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:22.247 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:22.304 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:28:22.355 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:28:22.355 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.355 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.356 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.356 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:28:22.358 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.358 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.358 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.359 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:28:22.391 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:22.430 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:22.431 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:28:22.487 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:28:22.487 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:28:22.487 INFO 2458 --- [ocp-schedules-5,88c16831325eb910,e9f108dfc1183bcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:28:22.619 INFO 2458 --- [ocp-async-3,25226f68f642af1d,1a2c82313c824f45] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:22.619 WARN 2458 --- [ocp-async-1,a2353f3499594df9,38f5e98a90c9989a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:23.220 INFO 2458 --- [host-check-clockdiff5,321dab46146b1326,4fff803dca6fec66] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535703 1 1 , error= 2025-10-27 11:28:23.707 INFO 2458 --- [prometheus-minute-schedule-1,6761f593dc9626aa,d792253e2fc81887] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:23.720 INFO 2458 --- [prometheus-minute-schedule-1,6761f593dc9626aa,d792253e2fc81887] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:24.448 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:24.449 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:24.619 WARN 2458 --- [ocp-async-5,fb39b7ead5286c04,b0f7b7587e8fdd42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:24.619 INFO 2458 --- [ocp-async-7,17a2b8949821048e,b8dc133dc7fb1bf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:25.217 INFO 2458 --- [ocp-schedules-28,f24549dd9535cdb7,6c18e9ebe28215cb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:25.221 INFO 2458 --- [ocp-schedules-28,f24549dd9535cdb7,6c18e9ebe28215cb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:26.619 WARN 2458 --- [ocp-async-2,8943ec2f6a54d42f,b9ff085878938137] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:26.619 INFO 2458 --- [ocp-async-6,abbb9964385f6d7c,b5c35c6dde49a0ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:26.707 INFO 2458 --- [prometheus-minute-schedule-1,674bcf96060be099,ebc71f9f83d0e67f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:26.722 INFO 2458 --- [prometheus-minute-schedule-1,674bcf96060be099,ebc71f9f83d0e67f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:26.844 INFO 2458 --- [ocp-schedules-30,496e976ad514b8b3,210bcf7e6f1ace8e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:27.449 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:27.450 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:27.501 INFO 2458 --- [ocp-schedules-26,af6dfab25c692bc8,bf57a4bbe0349389] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:28:27.508 INFO 2458 --- [ocp-schedules-26,af6dfab25c692bc8,bf57a4bbe0349389] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:27.508 INFO 2458 --- [ocp-schedules-26,af6dfab25c692bc8,bf57a4bbe0349389] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:28.530 INFO 2458 --- [ocp-schedules-30,e460266885d3fdcb,cbc8449776d87390] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:28.535 INFO 2458 --- [ocp-async-6,07cb872e36fc7416,c224e425c570dc2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:28.559 WARN 2458 --- [scanner-112194766-1,e460266885d3fdcb,5595d3b37999149f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:28.620 WARN 2458 --- [ocp-async-1,26f87c35e7592cb0,e63ae51ce826ca06] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:28.620 INFO 2458 --- [ocp-async-1,304b3d4fb9580a33,3b1596be6b51b4cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:28.682 INFO 2458 --- [alarm-detect-5,e460266885d3fdcb,ae12f70bb14993c9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535680 ~ 1761535698 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535695, value=72.744)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535695, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535695=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:28:29.015 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:28:29.015 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:28:29.019 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:28:29.019 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:28:29.020 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:28:29.020 INFO 2458 --- [ocp-schedules-20,ab9651d0a387e088,ca45f5c118514cf6] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:28:29.465 INFO 2458 --- [ocp-schedules-13,ac2fa25bf28c6d5b,4cf8854b71db6a72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:28:29.470 INFO 2458 --- [ocp-schedules-13,ac2fa25bf28c6d5b,4cf8854b71db6a72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:28:29.470 INFO 2458 --- [ocp-schedules-13,ac2fa25bf28c6d5b,4cf8854b71db6a72] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:28:29.707 INFO 2458 --- [prometheus-minute-schedule-1,c72ebe37a507a0a1,d1b5441a63c056d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:29.716 INFO 2458 --- [prometheus-minute-schedule-1,c72ebe37a507a0a1,d1b5441a63c056d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:30.450 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:30.451 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:30.620 WARN 2458 --- [ocp-async-4,6ef664486204a0f0,e01a2a8b2ebd4512] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:30.620 INFO 2458 --- [ocp-async-3,ba53a58b6120081e,48f2c7ce09277cf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:31.714246+08:00, lastProcessTime=2025-10-27T11:28:21.725933+08:00 2025-10-27 11:28:31.925 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:31.714246+08:00 2025-10-27 11:28:32.620 WARN 2458 --- [ocp-async-7,78dd092b82ab081e,9d0cbb2fe2e369f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:32.620 INFO 2458 --- [ocp-async-8,7a02700ac664c3ae,c238a29c31114029] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:32.707 INFO 2458 --- [prometheus-minute-schedule-1,91196959e3094263,4645c6d6687ce78e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:32.715 INFO 2458 --- [prometheus-minute-schedule-1,91196959e3094263,4645c6d6687ce78e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:33.451 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:33.452 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:33.931 INFO 2458 --- [ocp-schedules-30,dfc693db7fedff29,9e66aa8f52381761] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:28:33.931 INFO 2458 --- [ocp-schedules-30,dfc693db7fedff29,9e66aa8f52381761] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:28:34.620 WARN 2458 --- [ocp-async-9,b7fce79ec69868bb,44bb7572b18aa06b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:34.620 INFO 2458 --- [ocp-async-10,cc5f8c389b473923,847f67a5ad396180] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:35.228 INFO 2458 --- [ocp-schedules-18,5bd873203fdb40fa,8633c261e20ab46b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:35.232 INFO 2458 --- [ocp-schedules-18,5bd873203fdb40fa,8633c261e20ab46b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:35.707 INFO 2458 --- [prometheus-minute-schedule-1,a59eb877889db21d,26c760750d98f422] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:35.715 INFO 2458 --- [prometheus-minute-schedule-1,a59eb877889db21d,26c760750d98f422] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:36.452 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:36.453 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:36.620 INFO 2458 --- [ocp-async-10,bbcec3d6b31a06c8,d018901f6e2c5e7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:36.620 WARN 2458 --- [ocp-async-4,748e32827657d9e5,c2d387cd7beeab0c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:36.850 INFO 2458 --- [ocp-schedules-28,b9d4fe0af55b4e82,cf5bf101a4ec83ef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:37.708 INFO 2458 --- [prometheus-cache-monitor1,1f4189f4a98f1454,366cda7995555f6e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20387,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5097,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7886} 2025-10-27 11:28:37.708 INFO 2458 --- [prometheus-cache-monitor1,1f4189f4a98f1454,366cda7995555f6e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:28:38.536 INFO 2458 --- [ocp-async-2,66f8f681e90164d4,08ee4c2b972dd615] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:38.620 WARN 2458 --- [ocp-async-3,3f1e0d53dd29df18,43eb2598b3f2f7ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:38.620 INFO 2458 --- [ocp-async-7,5529aa70942c71da,f0e1fda0aec08648] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:38.687 INFO 2458 --- [ocp-schedules-2,d17d8bcb30b1098d,0ef8c8777064596a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:38.696 WARN 2458 --- [scanner-112194945-1,d17d8bcb30b1098d,03033ab86bc2c788] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:38.707 INFO 2458 --- [prometheus-minute-schedule-1,257ad70d01490e93,c00175b63de48be1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:38.717 INFO 2458 --- [prometheus-minute-schedule-1,257ad70d01490e93,c00175b63de48be1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:38.758 INFO 2458 --- [ocp-schedules-18,77ddcbefa25f415f,6d656a77c79080f6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:28:38.764 INFO 2458 --- [ocp-schedules-18,77ddcbefa25f415f,6d656a77c79080f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:38.768 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:28:38.768 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:28:38.803 INFO 2458 --- [ocp-schedules-18,77ddcbefa25f415f,6d656a77c79080f6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:28:39.145 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:28:39.145 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:28:39.222 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:39.222 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:39.261 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:39.261 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:39.261 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:39.262 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:28:39.275 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:39.275 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:39.275 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:39.276 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:28:39.343 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:39.344 INFO 2458 --- [ocp-schedules-23,9ee225bf7bccfae2,2958a13ba19594e1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:28:39.369 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:39.369 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:28:39.369 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:28:39.369 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:28:39.369 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:28:39.369 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:28:39.380 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:39.381 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:39.381 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:39.381 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:39.381 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:39.382 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:39.384 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:39.385 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:39.385 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:39.385 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:39.453 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:39.454 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:39.508 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:28:39.512 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:28:39.514 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:28:39.519 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:28:39.525 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:28:39.527 INFO 2458 --- [ocp-schedules-24,7bcfa24decfe8d72,63e7a15c197578dd] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:28:40.014 INFO 2458 --- [ocp-schedules-12,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:28:40.015 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:28:40.061 INFO 2458 --- [Check obproxy1,67b07b656b799b79,9d6fb6145fba6225] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:28:40.061 INFO 2458 --- [Check obproxy1,67b07b656b799b79,9d6fb6145fba6225] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:28:40.062 INFO 2458 --- [Collect obproxy1,60900c8d89b72168,39dd9177b20716b3] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:28:40.063 INFO 2458 --- [Collect obproxy1,60900c8d89b72168,39dd9177b20716b3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:28:40.091 INFO 2458 --- [Check obproxy1,67b07b656b799b79,9d6fb6145fba6225] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:28:40.092 INFO 2458 --- [Collect obproxy1,60900c8d89b72168,39dd9177b20716b3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:28:40.095 INFO 2458 --- [Check obproxy1,67b07b656b799b79,9d6fb6145fba6225] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:28:40.095 INFO 2458 --- [Check obproxy1,67b07b656b799b79,9d6fb6145fba6225] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:28:40.095 INFO 2458 --- [Collect obproxy1,60900c8d89b72168,39dd9177b20716b3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:28:40.098 INFO 2458 --- [Collect obproxy1,60900c8d89b72168,39dd9177b20716b3] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:28:40.620 INFO 2458 --- [ocp-async-9,171c088f03304575,0b8d5809d2f9a6da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:40.620 WARN 2458 --- [ocp-async-8,b33bda4f4bad5a5e,3d4e9214d12bf6a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:41.262 INFO 2458 --- [ocp-schedules-20,08abf3e008bb3e53,99d2732e204a4152] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=1 2025-10-27 11:28:41.263 INFO 2458 --- [ocp-schedules-20,08abf3e008bb3e53,99d2732e204a4152] c.o.o.b.i.BinlogInstanceSyncService : [OCP811] sync binlog instances begin, binlogClusterId=4 2025-10-27 11:28:41.707 INFO 2458 --- [prometheus-minute-schedule-1,84501a269bf84551,c7be45685d496ba3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:41.713 INFO 2458 --- [prometheus-minute-schedule-1,84501a269bf84551,c7be45685d496ba3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:41.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:41.719017+08:00, lastProcessTime=2025-10-27T11:28:31.714246+08:00 2025-10-27 11:28:41.939 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:41.719017+08:00 2025-10-27 11:28:42.454 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:42.455 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:42.621 WARN 2458 --- [ocp-async-1,b7be1a9c0d778665,dd17271d6d4308e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:42.621 INFO 2458 --- [ocp-async-4,70c168e9f212ef3c,73a48c4cc9c6b798] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:44.229 INFO 2458 --- [ocp-schedules-14,fec2b130cc72ccb3,4570acc96ead3da3] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:28:44.621 WARN 2458 --- [ocp-async-2,6bebbffb9baed601,1a823947a80ecbc0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:44.621 INFO 2458 --- [ocp-async-2,5a7c4ffd34c4a697,295be6d2570798cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:44.707 INFO 2458 --- [prometheus-minute-schedule-1,90807432b9a9031e,3d97c90c06184522] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:44.715 INFO 2458 --- [prometheus-minute-schedule-1,90807432b9a9031e,3d97c90c06184522] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:45.102 INFO 2458 --- [prometheus-second-schedule-1,2ffb516b4175a299,d7940e44a7fc1cf4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=16, collectTaskCount=16 2025-10-27 11:28:45.240 INFO 2458 --- [ocp-schedules-23,857343c984d0fd77,b36ebdd2d04272cb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:45.244 INFO 2458 --- [ocp-schedules-23,857343c984d0fd77,b36ebdd2d04272cb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:45.455 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:45.456 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:46.449 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [DELETE /api/v2/binlog/clusters/4, client 10.40.198.13, traceId 90c99b6302573800] 2025-10-27 11:28:46.463 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.o.s.c.trace.RequestTracingAspect : API: [DELETE /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=90c99b6302573800, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.deleteBinlogCluster(Long), args=4,] 2025-10-27 11:28:46.485 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Delete binlog cluster, nodeCount=11, taskCount=11 2025-10-27 11:28:46.485 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Delete binlog cluster', taskDefinitionId=-1, state=RUNNING, type=MANUAL, clusterId=null, tenantId=null, hostId=null, creator='admin', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, prohibit_rollback=true, service_name=binlogtest2, install_path=/home/admin/binlogservice, service_version=4.0.1-100000162024122511, binlog_cluster_id=4}, listMap={binlog_server_ids=[4], host_ids=[4]}}, createTime=null, startTime=2025-10-27T11:28:46.485+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Stop binlog instance process, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.StopBinlogInstanceProcessTask, seriesId=5, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Wait node, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.WaitNode, seriesId=9, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Unreserve host, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.host.UnReserveHostTask, seriesId=1, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Delete binlog cluster metadb, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.DeleteBinlogClusterMetadbTask, seriesId=4, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Stop binlog server, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.StopBinlogServerTask, seriesId=2, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Clean binlog server, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.binlog.internal.task.CleanBinlogServerTask, seriesId=0, startTime=null, endTime=null}, SubtaskInstanceEntity{id=null, name=Inactive agent modules, state=PENDING, operation=EXECUTE, className=com.oceanbase.ocp.service.task.business.obagent.InactiveAgen 2025-10-27 11:28:46.507 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=9169} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=4, svrIp='10.40.65.151', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-10-27 11:28:46.521 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=9165} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=4, svrIp='10.40.65.151', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-10-27 11:28:46.522 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=9171} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=4, svrIp='10.40.65.151', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-10-27 11:28:46.523 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.task.TaskTagService : Save subtask tags, tag=SubtaskInstanceTagEntity{subtaskId=9170} BaseTagEntity{id=null, clusterName='null', clusterId=null, obClusterId=null, zoneName='null', tenantName='null', tenantId=null, obTenantId=null, serverId=null, obproxyClusterName='null', obproxyClusterId=null, obproxyServerId=null, arbitrationServiceId=null, arbitrationServicePort=null, hostId=4, svrIp='10.40.65.151', deleted=false, createTime=null, binlogServerId=null, binlogClusterName=null, binlogClusterId=null, tenantBinlogServiceId=null, binlogInstanceName=null} 2025-10-27 11:28:46.524 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.task.TaskTagService : Save task tags, taskId=9107, taskName=Delete binlog cluster, argument=Argument{parallelIdx=-1, stringMap={service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, prohibit_rollback=true, service_name=binlogtest2, install_path=/home/admin/binlogservice, service_version=4.0.1-100000162024122511, binlog_cluster_id=4}, listMap={binlog_server_ids=[4], host_ids=[4]}} 2025-10-27 11:28:46.539 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [DELETE /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=90c99b6302573800, duration=117 ms] 2025-10-27 11:28:46.539 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,90c99b6302573800,4a3347160a55e87f] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=DELETE_BINLOG_CLUSTER, eventType=BINLOG_SERVICE_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.BinlogController.deleteBinlogCluster, resourceType=BINLOG_SERVICE, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=4, resourceName=binlogtest2, parentResource=null)], operatorId=100, operatorName=admin, status=UNFINISHED, failedReason=null, taskId=9107, eventInfo={"clusterId":4}, executionType=NEW, requestUrl=http://10.40.198.13:24631/api/v2/binlog/clusters/4, clientIp=10.40.198.13, serverIp=10.40.65.146, startTime=2025-10-27T11:28:46.463+08:00, endTime=null, role=null, aliUid=null) 2025-10-27 11:28:46.601 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,861dac44a6c30584] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9174, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:46.621 WARN 2458 --- [ocp-async-2,458ff21946f821f4,0207ad528839d0d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:46.621 INFO 2458 --- [ocp-async-2,ff47c002d6c5c306,2a383c6f4e876604] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:46.640 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:46.646 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9174, context=Context{parallelIdx=-1, stringMap={service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, latest_execution_start_time=2025-10-27T11:28:46.586+08:00, sub_task_instance_name=Record binlog instance name, sub_task_instance_id=9174, binlog_cluster_id=4}, listMap={binlog_server_ids=[4], host_ids=[4]}}, executor=10.40.65.146 2025-10-27 11:28:46.655 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:28:46.663 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:28:46.779 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,37bc3534c52ebadd,efe1f6bb23e2bfd8] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clusters/4?clusterId=4, client=10.40.198.13, traceId=37bc3534c52ebadd, method=SuccessResponse com.oceanbase.ocp.server.common.controller.BinlogController.getBinlogCluster(Long), args=4,] 2025-10-27 11:28:46.779 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,88844ab914fd632f,5eda07f046365713] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=88844ab914fd632f, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:46.780 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,9c4778ec18af40fb,820d68ec64e8cd14] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=9c4778ec18af40fb, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:28:46.783 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,9c4778ec18af40fb,820d68ec64e8cd14] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=9c4778ec18af40fb, duration=13 ms] 2025-10-27 11:28:46.807 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,37bc3534c52ebadd,efe1f6bb23e2bfd8] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:28:46.807 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,37bc3534c52ebadd,efe1f6bb23e2bfd8] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW BINLOG INSTANCES 2025-10-27 11:28:46.795 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.o.b.i.t.RecordBinlogInstanceNameTask : Success to put binlog instances names of cluster to context: [] 2025-10-27 11:28:46.817 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9174, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:46.826 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,88d202c5f5211ccb] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9169] 2025-10-27 11:28:46.858 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,88844ab914fd632f,5eda07f046365713] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=88844ab914fd632f, duration=88 ms] 2025-10-27 11:28:46.860 INFO 2458 --- [ocp-schedules-17,02aca5d3cfbffdb1,bc2a7cea5442e62a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:46.936 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,37bc3534c52ebadd,efe1f6bb23e2bfd8] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clusters/4 client=10.40.198.13, traceId=37bc3534c52ebadd, duration=176 ms] 2025-10-27 11:28:46.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.b.operator.JdbcBinlogOperator : binlog operator created, connectProperties=ConnectProperties(address=10.40.65.151, port=2983, username=null, password=null) 2025-10-27 11:28:46.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.binlogsdk.operator.BinlogTemplate : [binlogsdk] sql: SHOW NODES 2025-10-27 11:28:47.511 INFO 2458 --- [ocp-schedules-9,25fb5a1305b568f2,3eeaa44f062696a7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:28:47.519 INFO 2458 --- [ocp-schedules-9,25fb5a1305b568f2,3eeaa44f062696a7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:47.519 INFO 2458 --- [ocp-schedules-9,25fb5a1305b568f2,3eeaa44f062696a7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:28:47.623 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,bb403fdb7507f749] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9169, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:47.651 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:47.657 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9169, context=Context{parallelIdx=0, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:47.621+08:00, sub_task_instance_name=Stop binlog server, sub_task_instance_id=9169}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:47.665 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.b.i.task.StopBinlogServerTask : stop binlog server... 2025-10-27 11:28:47.688 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:47.696 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:47.707 INFO 2458 --- [prometheus-minute-schedule-1,417b94707eb3c1b1,08f288040e5a3d23] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:47.714 INFO 2458 --- [prometheus-minute-schedule-1,417b94707eb3c1b1,08f288040e5a3d23] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:47.768 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/stop, request body:StopProcessRequest(process=FindProcessParam(findType=BY_NAME, name=logproxy, keyword=null, pid=null), force=true), params:null 2025-10-27 11:28:47.780 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.b.i.task.StopBinlogServerTask : wait binlog server stopped... 2025-10-27 11:28:47.800 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:47.808 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:47.865 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/exists, request body:CheckProcessExistsRequest(name=logproxy), params:null 2025-10-27 11:28:47.873 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.b.internal.BinlogOperationManager : logproxy process is not exist. 2025-10-27 11:28:47.878 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.b.internal.BinlogOperationManager : binlog server (id=4) process stopped 2025-10-27 11:28:47.890 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9169, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:47.893 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,da619cd97a6b4abc] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9175] 2025-10-27 11:28:48.085 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,2d9c3a4e2b46a0cf,4e6e58b9e6d79b92] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=2d9c3a4e2b46a0cf, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:48.100 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,2d9c3a4e2b46a0cf,4e6e58b9e6d79b92] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=2d9c3a4e2b46a0cf, duration=24 ms] 2025-10-27 11:28:48.456 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:48.457 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:48.536 INFO 2458 --- [ocp-async-2,09c032d76fac9ef5,cad623de949b939f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:48.621 WARN 2458 --- [ocp-async-10,f1533e20f6c9d181,2c0863799751eb13] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:48.621 INFO 2458 --- [ocp-async-1,aeb726b6c5e63066,43aa63ded1175120] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:48.640 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,0bc6a6ca6c02a11d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9175, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:48.681 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,1a47ae47db0400a9] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:48.688 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,1a47ae47db0400a9] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9175, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:48.638+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=9175}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:48.704 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,1a47ae47db0400a9] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9175, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:48.706 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,1a47ae47db0400a9] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9165] 2025-10-27 11:28:48.875 INFO 2458 --- [ocp-schedules-13,f536edae541e8d7d,b807fc07af931579] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:48.995 WARN 2458 --- [scanner-112195187-1,f536edae541e8d7d,666d6fec7b7127ab] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:49.015 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?backUrl=/task, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:28:49.015 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/task/9107, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:28:49.015 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:28:49.095 INFO 2458 --- [alarm-detect-6,f536edae541e8d7d,e1f216d34a373a37] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535695 ~ 1761535719 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535710, value=109.527)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535710, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535710=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:28:49.501 INFO 2458 --- [ocp-schedules-2,af2e85c8a55b7218,c32e5c7d796902c7] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:28:49.501+08:00, clusters=[1, 2] 2025-10-27 11:28:49.531 INFO 2458 --- [ocp-schedules-2,af2e85c8a55b7218,c32e5c7d796902c7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:28:49.534 INFO 2458 --- [ocp-schedules-2,af2e85c8a55b7218,c32e5c7d796902c7] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:28:49.613 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ed607d9508618019,aada4a81fb03937a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=ed607d9508618019, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:49.628 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ed607d9508618019,aada4a81fb03937a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=ed607d9508618019, duration=25 ms] 2025-10-27 11:28:49.666 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,168acaab85362fd7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9165, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:49.699 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,506afe0e3efe1924] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:49.704 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,506afe0e3efe1924] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9165, context=Context{parallelIdx=0, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:49.664+08:00, sub_task_instance_name=Stop binlog instance process, sub_task_instance_id=9165}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:49.709 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,506afe0e3efe1924] .o.o.b.i.t.StopBinlogInstanceProcessTask : Do not have instance names in context, maybe the binlog cluster do not have any instances, or subtask [Record binlog instance name] failed to record it. 2025-10-27 11:28:49.721 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,506afe0e3efe1924] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9165, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:49.723 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,506afe0e3efe1924] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9166] 2025-10-27 11:28:49.912 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3406e37bb2a5abfb,6e54ea9308657f29] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=3406e37bb2a5abfb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:49.922 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,8d5491f0aeecb8d3,a2f1d638e0ee3a4b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=8d5491f0aeecb8d3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:28:49.924 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fd8572a6f49d3a41,a2c6e001f5c6fbaa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/authInfo?null, client=10.40.198.13, traceId=fd8572a6f49d3a41, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getAuthInfo(), args=] 2025-10-27 11:28:49.929 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,e1f7352034a503c2,3f88deccfc0772cb] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=e1f7352034a503c2, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:28:49.929 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,e1f7352034a503c2,3f88deccfc0772cb] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=e1f7352034a503c2, duration=15 ms] 2025-10-27 11:28:49.942 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,fd8572a6f49d3a41,a2c6e001f5c6fbaa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/authInfo client=10.40.198.13, traceId=fd8572a6f49d3a41, duration=28 ms] 2025-10-27 11:28:49.946 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:28:49.946 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:28:49.946 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:28:49.946 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,8d5491f0aeecb8d3,a2f1d638e0ee3a4b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=8d5491f0aeecb8d3, duration=32 ms] 2025-10-27 11:28:49.969 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,3406e37bb2a5abfb,6e54ea9308657f29] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=3406e37bb2a5abfb, duration=66 ms] 2025-10-27 11:28:50.036 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,761204f4c9f795d1,c268247856ca73e2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=761204f4c9f795d1, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:28:50.160 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,761204f4c9f795d1,c268247856ca73e2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=761204f4c9f795d1, duration=131 ms] 2025-10-27 11:28:50.283 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6d5c5ca14f0f38d3,3f3a9afc6d061b3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=6d5c5ca14f0f38d3, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:28:50.320 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b5290e8aedd093ae,af66e21458aaed89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107/subtasks/9166/logSegments?taskInstanceId=9107&subtaskInstanceId=9166, client=10.40.198.13, traceId=b5290e8aedd093ae, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9107,9166,] 2025-10-27 11:28:50.329 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,b5290e8aedd093ae,af66e21458aaed89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107/subtasks/9166/logSegments client=10.40.198.13, traceId=b5290e8aedd093ae, duration=17 ms] 2025-10-27 11:28:50.516 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6d5c5ca14f0f38d3,3f3a9afc6d061b3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=6d5c5ca14f0f38d3, duration=243 ms] 2025-10-27 11:28:50.621 WARN 2458 --- [ocp-async-6,f8336ed66d7534cb,0cbfd520460fc8f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:50.621 INFO 2458 --- [ocp-async-3,a0a5c12abc076531,fb9e4456e6567c24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:50.682 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,dbc2740ca6e8c0f8] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9166, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:50.707 INFO 2458 --- [prometheus-minute-schedule-1,7c98ef8ae30103cd,5b4ac234ca1a6716] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:50.707 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,0679a5a82422f983] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:50.714 INFO 2458 --- [prometheus-minute-schedule-1,7c98ef8ae30103cd,5b4ac234ca1a6716] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:50.713 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,0679a5a82422f983] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9166, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:50.680+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=9166}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:50.728 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,0679a5a82422f983] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9166, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:50.730 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,0679a5a82422f983] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9170] 2025-10-27 11:28:51.123 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d41817f4347ef085,bbd4109181d8bdb2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=d41817f4347ef085, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:51.143 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d41817f4347ef085,bbd4109181d8bdb2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=d41817f4347ef085, duration=29 ms] 2025-10-27 11:28:51.205 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,e1f542bac855aa34,e19f93f2bd84f98a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107/subtasks/9166/logSegments?taskInstanceId=9107&subtaskInstanceId=9166, client=10.40.198.13, traceId=e1f542bac855aa34, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getSubtaskLogSegment(Long,Long), args=9107,9166,] 2025-10-27 11:28:51.219 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,e1f542bac855aa34,e19f93f2bd84f98a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107/subtasks/9166/logSegments client=10.40.198.13, traceId=e1f542bac855aa34, duration=22 ms] 2025-10-27 11:28:51.408 INFO 2458 --- [ocp-schedules-30,1b45b24403cdfaa1,2c1863d3aae6f6da] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:51.409 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:28:51.409 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:28:51.409 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:28:51.409 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:28:51.409 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:28:51.420 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:51.421 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:51.423 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:51.424 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:51.426 INFO 2458 --- [ocp-schedules-5,284a8eb1dd658b58,d9aea623ba20d213] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:51.426 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:51.426 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:28:51.426 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:51.427 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:28:51.427 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:28:51.427 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=ONLINE 2025-10-27 11:28:51.427 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:51.427 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:51.428 INFO 2458 --- [ocp-schedules-19,288853796b737e3e,bda5490780f8e8d9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:28:51.428 INFO 2458 --- [host-check-clockdiff1,288853796b737e3e,9385204971630898] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:28:51.428 INFO 2458 --- [host-check-clockdiff2,288853796b737e3e,a17d7e2b1fff70d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:28:51.428 INFO 2458 --- [host-check-clockdiff3,288853796b737e3e,cca1bc50939f16bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:28:51.429 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:51.430 INFO 2458 --- [host-check-clockdiff4,288853796b737e3e,b5793fb49be1e182] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:28:51.430 INFO 2458 --- [host-check-clockdiff5,288853796b737e3e,eaf3552c1e0ae7c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:28:51.430 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:28:51.431 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:51.439 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:51.441 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:51.442 INFO 2458 --- [ocp-schedules-24,3489bb7ca81fa6ed,ef80f4a6bcf49c79] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:28:51.442 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:51.442 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:51.443 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:51.443 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:51.463 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:51.463 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:51.464 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:51.464 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:51.464 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:51.464 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:51.477 INFO 2458 --- [host-check-clockdiff2,288853796b737e3e,a17d7e2b1fff70d3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535731 -1 -1 , error= 2025-10-27 11:28:51.481 INFO 2458 --- [host-check-clockdiff3,288853796b737e3e,cca1bc50939f16bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535731 -1 -1 , error= 2025-10-27 11:28:51.487 INFO 2458 --- [host-check-clockdiff5,288853796b737e3e,eaf3552c1e0ae7c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535731 0 0 , error= 2025-10-27 11:28:51.487 INFO 2458 --- [host-check-clockdiff1,288853796b737e3e,9385204971630898] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535731 0 0 , error= 2025-10-27 11:28:51.519 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:28:51.521 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:28:51.521 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:28:51.520180175+08:00 2025-10-27 11:28:51.522 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:28:51.521+08:00 2025-10-27 11:28:51.524 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:28:51.523657339+08:00 2025-10-27 11:28:51.524 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:28:51.524+08:00 2025-10-27 11:28:51.526 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:28:51.571 INFO 2458 --- [ocp-schedules-8,7cc531b1e4a9d487,f43ec7d498fcf95a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:28:51.571 INFO 2458 --- [ocp-schedules-21,e904c1728b98c09d,f9610d592a756c59] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:28:51.571 INFO 2458 --- [ocp-schedules-7,2ed6baf825c92394,38388a8d3980fac7] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:28:51.571 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:28:51.524+08:00 and status AVAILABLE 2025-10-27 11:28:51.572 INFO 2458 --- [ocp-schedules-23,b790a1aa4dd596a4,32ddb6053850402e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:28:51.573 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:28:51.521+08:00 and status AVAILABLE 2025-10-27 11:28:51.575 INFO 2458 --- [ocp-schedules-15,2111304ba1831752,b2d78150a7a0d850] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:28:51.577 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:28:51.581 INFO 2458 --- [ocp-schedules-7,2ed6baf825c92394,38388a8d3980fac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:51.583 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:28:51.586 INFO 2458 --- [ocp-schedules-17,73745e851247618c,dbcd9135bea5022e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:28:51.586 INFO 2458 --- [ocp-schedules-8,7cc531b1e4a9d487,f43ec7d498fcf95a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:51.586 INFO 2458 --- [ocp-schedules-21,e904c1728b98c09d,f9610d592a756c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:28:51.587 INFO 2458 --- [ocp-schedules-15,2111304ba1831752,b2d78150a7a0d850] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:28:51.588 INFO 2458 --- [ocp-schedules-23,b790a1aa4dd596a4,32ddb6053850402e] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:28:51.590 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:28:51.590 INFO 2458 --- [ocp-schedules-21,e904c1728b98c09d,f9610d592a756c59] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:28:51.599 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:28:51.598486228+08:00 2025-10-27 11:28:51.600 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:28:51.599+08:00 2025-10-27 11:28:51.604 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,3a60abc92f0a0d94,857f56a41d77e1aa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=3a60abc92f0a0d94, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:51.606 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:28:51.527915413+08:00 2025-10-27 11:28:51.606 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:28:51.607 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:28:51.606+08:00 2025-10-27 11:28:51.608 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:28:51.609 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:28:51.609 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:28:51.609 INFO 2458 --- [host-check-agent-status5,284a8eb1dd658b58,42121dad9fb63309] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:28:51.612 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:28:51.612 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:28:51.612 INFO 2458 --- [host-check-agent-status4,284a8eb1dd658b58,9048d34d87719ca1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:28:51.613 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:28:51.613 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:28:51.599+08:00 and status AVAILABLE 2025-10-27 11:28:51.614 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:28:51.617 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:28:51.617 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:28:51.617 INFO 2458 --- [host-check-agent-status1,284a8eb1dd658b58,4e1d92c5beecdd61] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:28:51.618 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:28:51.618 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:28:51.618 INFO 2458 --- [host-check-agent-status3,284a8eb1dd658b58,cc439f731618f857] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:28:51.618 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:28:51.622 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:28:51.606+08:00 and status AVAILABLE 2025-10-27 11:28:51.622 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:28:51.623 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:51.625 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:28:51.626 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:51.626 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:28:51.625623612+08:00 2025-10-27 11:28:51.626 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:28:51.627 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:28:51.627 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:28:51.626+08:00 2025-10-27 11:28:51.628 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:28:51.631 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:28:51.631 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:28:51.631 INFO 2458 --- [host-check-agent-status2,284a8eb1dd658b58,9a4299624fbb5068] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:28:51.632 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,3a60abc92f0a0d94,857f56a41d77e1aa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=3a60abc92f0a0d94, duration=61 ms] 2025-10-27 11:28:51.637 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:28:51.626+08:00 and status AVAILABLE 2025-10-27 11:28:51.642 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:28:51.657 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:51.661 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:51.676 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.676 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:51.678 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:28:51.678 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:28:51.680 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:51.681 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:28:51.683 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:51.692 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.692 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.692 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.692 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:51.692 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.692 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.693 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.693 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:51.701 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:28:51.701 INFO 2458 --- [ocp-schedules-8,7cc531b1e4a9d487,f43ec7d498fcf95a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:28:51.712 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,9e1b463542d4f170] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9170, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:51.723 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:28:51.721732+08:00, lastProcessTime=2025-10-27T11:28:41.719017+08:00 2025-10-27 11:28:51.747 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:51.753 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:51.762 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:51.777 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:28:51.777 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:28:51.772 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:51.779 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:28:51.779 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9170, context=Context{parallelIdx=0, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:51.703+08:00, sub_task_instance_name=Clean binlog server, sub_task_instance_id=9170}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:51.789 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:51.793 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:51.793 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:28:51.785 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall, hostId=4, packageName=obbinlog 2025-10-27 11:28:51.794 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:28:51.807 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:28:51.807 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.807 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.808 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.808 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.808 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.808 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:51.813 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:28:51.806 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:51.816 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:28:51.831 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:28:51.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,2e62b8dbbbe42116,4d764b4548fe6c4d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=2e62b8dbbbe42116, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:28:51.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,2e62b8dbbbe42116,4d764b4548fe6c4d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=2e62b8dbbbe42116, duration=12 ms] 2025-10-27 11:28:51.832 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:28:51.817 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:51.837 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,e779ee8ac7363917,2a6b89a3c847bc36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=e779ee8ac7363917, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:28:51.837 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,e779ee8ac7363917,2a6b89a3c847bc36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=e779ee8ac7363917, duration=18 ms] 2025-10-27 11:28:51.845 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,4abe23bd4847dfa9,bd55a13d65e8f4d2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=4abe23bd4847dfa9, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:28:51.845 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,4abe23bd4847dfa9,bd55a13d65e8f4d2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=4abe23bd4847dfa9, duration=26 ms] 2025-10-27 11:28:51.863 INFO 2458 --- [ocp-schedules-16,0ef68d8ac2c1c0c8,97fd51e973961c18] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:28:51.867 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:28:51.870 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:28:51.870 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:28:51.870 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:28:51.872 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:28:51.874 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:51.878 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:51.879 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:51.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:51.885 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.886 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:51.887 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.890 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:28:51.890 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:28:51.891 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:51.899 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:28:51.908 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:51.914 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:28:51.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:51.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:51.916 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:51.921 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.922 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:28:51.920 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/package/uninstall, request body:UninstallPackageRequest(name=obbinlog, installPath=null), params:null 2025-10-27 11:28:51.927 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:28:51.932 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:28:51.938 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:28:51.940 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:28:51.947 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:51.959 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:28:51.959 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:28:51.965 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:51.968 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:51.975 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:28:51.978 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:51.980 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:28:51.980 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:28:51.989 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:28:51.992 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:51.993 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:28:51.995 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:52.003 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:28:52.005 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:28:52.008 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:28:52.009 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:28:52.013 INFO 2458 --- [host-check-host-status4,1b45b24403cdfaa1,a5324932eb4b8f2e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:28:52.018 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:52.018 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.018 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.019 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.019 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.019 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.020 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:28:52.021 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.021 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.021 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:28:52.025 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:28:52.025 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:52.026 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:28:52.029 INFO 2458 --- [host-check-host-status2,1b45b24403cdfaa1,fe4207384df6fb8c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:28:52.054 INFO 2458 --- [host-check-host-status1,1b45b24403cdfaa1,2c43d8d7d94b5603] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:28:52.054 INFO 2458 --- [host-check-host-status5,1b45b24403cdfaa1,64e5e6c4d28351bb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:28:52.057 INFO 2458 --- [host-check-host-status3,1b45b24403cdfaa1,efaefed77ebd2dca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:28:52.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:52.080 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:28:51.721732+08:00 2025-10-27 11:28:52.083 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:28:52.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:52.125 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:28:52.125 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.126 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:28:52.128 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.129 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:28:52.144 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:28:52.149 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.149 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:52.170 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:52.170 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.171 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:28:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:28:52.193 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:28:52.193 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:28:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.223 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:28:52.224 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,cbe303fd9cb1141e,623a76b776bced5c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=cbe303fd9cb1141e, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:52.244 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,cbe303fd9cb1141e,623a76b776bced5c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=cbe303fd9cb1141e, duration=29 ms] 2025-10-27 11:28:52.263 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:28:52.300 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:52.300 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.300 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.301 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.302 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:28:52.305 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:28:52.323 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:28:52.332 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:28:52.335 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:28:52.383 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:28:52.430 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:28:52.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.430 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.430 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.431 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:28:52.434 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.434 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.434 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.435 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:28:52.475 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:28:52.515 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:52.515 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.515 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.516 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:52.516 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:52.516 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:52.516 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:28:52.558 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:28:52.558 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:28:52.558 INFO 2458 --- [ocp-schedules-7,2ed6baf825c92394,38388a8d3980fac7] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:28:52.621 WARN 2458 --- [ocp-async-2,1eab2582b4f9942f,06bfe244a95c14a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:52.621 INFO 2458 --- [ocp-async-2,87ee0f215d41fa72,f557933b6526122f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:52.967 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall do nothing, package not exists 2025-10-27 11:28:52.997 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall done 2025-10-27 11:28:53.012 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall, hostId=4, packageName=obbinlog-ce 2025-10-27 11:28:53.039 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:53.049 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:53.109 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/package/uninstall, request body:UninstallPackageRequest(name=obbinlog-ce, installPath=null), params:null 2025-10-27 11:28:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:28:53.323 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,db91919fa0f18117,0c49c3a25581f60e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=db91919fa0f18117, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:53.346 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,db91919fa0f18117,0c49c3a25581f60e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=db91919fa0f18117, duration=32 ms] 2025-10-27 11:28:53.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,922df8282153eca9,e2c730bc4484044e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=922df8282153eca9, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:53.575 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,922df8282153eca9,e2c730bc4484044e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=922df8282153eca9, duration=25 ms] 2025-10-27 11:28:53.707 INFO 2458 --- [prometheus-minute-schedule-1,3e929f4bef3d9c77,1f005cac487ea481] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:53.714 INFO 2458 --- [prometheus-minute-schedule-1,3e929f4bef3d9c77,1f005cac487ea481] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:53.760 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.h.s.HostOperationServiceImpl : rpm uninstall done 2025-10-27 11:28:53.767 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.b.i.BinlogHostOperationService : [cleanAllFiles]: generate agent async task token:dab9bfeb-be1d-4715-bf2f-911384868888 2025-10-27 11:28:53.783 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:53.791 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:53.844 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/binlog/cleanAllFiles, request body:CleanBinlogFilesRequest(installPath=/home/admin/binlogservice), params:null 2025-10-27 11:28:53.861 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/task/status, request body:GetTaskStatusRequest(taskToken=dab9bfeb-be1d-4715-bf2f-911384868888), params:null 2025-10-27 11:28:53.868 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.s.task.util.AgentAsyncTaskHelper : try to request task result(EXECUTE), result:true,null,, 2025-10-27 11:28:53.879 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9170, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:53.882 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,5186aaf73e054e32] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9173] 2025-10-27 11:28:53.956 INFO 2458 --- [host-check-clockdiff4,288853796b737e3e,b5793fb49be1e182] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535733 0 0 , error= 2025-10-27 11:28:54.423 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a3414172b8f3df68,01da0dcb45687f4c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=a3414172b8f3df68, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:54.445 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a3414172b8f3df68,01da0dcb45687f4c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=a3414172b8f3df68, duration=32 ms] 2025-10-27 11:28:54.464 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:54.465 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:54.599 INFO 2458 --- [ocp-schedules-17,238f70006a2a2335,f65b9218acda1cb5] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-10-27 11:28:54.599 INFO 2458 --- [ocp-schedules-17,238f70006a2a2335,f65b9218acda1cb5] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-10-27 11:28:54.621 INFO 2458 --- [ocp-async-10,7b88b699e7e67178,b0cc324f6eb5a67d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:54.622 WARN 2458 --- [ocp-async-1,0c1ff2fd2815a815,708fde3d05e55a0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:54.739 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,7a0626fa4cc7fd1d] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9173, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:54.765 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,17061060e07161be] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:54.770 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,17061060e07161be] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9173, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:54.736+08:00, sub_task_instance_name=Wait node, sub_task_instance_id=9173}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:54.788 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,17061060e07161be] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9173, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:54.793 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,17061060e07161be] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9168] 2025-10-27 11:28:54.818 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,f25cd1958e832bee,43ce35152272b0b3] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=f25cd1958e832bee, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:28:54.831 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,f25cd1958e832bee,43ce35152272b0b3] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=f25cd1958e832bee, duration=21 ms] 2025-10-27 11:28:54.840 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,eddf6b1d0e8b1ecb,85405407ca47cafc] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=eddf6b1d0e8b1ecb, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:28:54.840 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,eddf6b1d0e8b1ecb,85405407ca47cafc] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=eddf6b1d0e8b1ecb, duration=9 ms] 2025-10-27 11:28:55.252 INFO 2458 --- [ocp-schedules-6,93b1180ab7302963,2ba0bc8ed883fe92] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:28:55.256 INFO 2458 --- [ocp-schedules-6,93b1180ab7302963,2ba0bc8ed883fe92] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:28:55.420 INFO 2458 --- [ocp-schedules-23,861647005efd4932,0c455031320aea11] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:28:55.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,0231696c9268cd72,e4ce4a4e1397f0a1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=0231696c9268cd72, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:55.546 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,0231696c9268cd72,e4ce4a4e1397f0a1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=0231696c9268cd72, duration=30 ms] 2025-10-27 11:28:55.569 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,cbb43c60c8dd9b11,d978e0e3a8a27341] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=cbb43c60c8dd9b11, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:55.583 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,cbb43c60c8dd9b11,d978e0e3a8a27341] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=cbb43c60c8dd9b11, duration=22 ms] 2025-10-27 11:28:55.757 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,3696977725e4672a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9168, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:55.781 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:55.789 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9168, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:55.753+08:00, sub_task_instance_name=Delete binlog cluster metadb, sub_task_instance_id=9168}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:55.918 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=ocp_meta, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:55.926 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=ocp_meta, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:55.931 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=ocp_meta, username=root, address=10.40.65.146, port=2881, database=oceanbase) 2025-10-27 11:28:55.936 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector, connectProperties=ConnectProperties(connectionMode=direct, address=null, port=null, obsAddrList=[ObServerAddr(address=10.40.65.168, port=2881), ObServerAddr(address=10.40.67.22, port=2881), ObServerAddr(address=10.40.65.146, port=2881)], username=root, tenantName=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-10-27 11:28:55.949 INFO 2458 --- [obsdk-init-thread-2,90c99b6302573800,873eb756535c168f] com.alibaba.druid.pool.DruidDataSource : {dataSource-22,ob-connector-datasource_direct_ocp_meta_root_10.40.65.168_2881} inited 2025-10-27 11:28:55.949 INFO 2458 --- [obsdk-init-thread-2,90c99b6302573800,873eb756535c168f] c.o.o.o.c.impl.AbstractConnector : [obsdk] init druid datasource success, poolingCount=1, connectProperties=ConnectProperties(connectionMode=direct, address=10.40.65.168, port=2881, obsAddrList=[ObServerAddr(address=10.40.65.168, port=2881), ObServerAddr(address=10.40.67.22, port=2881), ObServerAddr(address=10.40.65.146, port=2881)], username=root, tenantName=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, proxy=null, odpSocketConfig=null, vpcId=1, compatibilityMode=MYSQL, database=oceanbase) 2025-10-27 11:28:55.950 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ObConnectors : [obsdk] create new ob connector success 2025-10-27 11:28:55.957 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:55.963 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW DATABASES 2025-10-27 11:28:55.994 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:56.003 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ o.CREATED AS GMT_CREATE, o.OBJECT_ID AS DATABASE_ID, d.DATABASE_NAME, c.ID AS COLLATION_TYPE, NULL AS PRIMARY_ZONE, (CASE READ_ONLY WHEN 'YES' THEN 1 ELSE 0 END) AS READ_ONLY FROM oceanbase.DBA_OB_DATABASES d JOIN oceanbase.DBA_OBJECTS o JOIN information_schema.collations c ON BINARY d.DATABASE_NAME = BINARY o.OBJECT_NAME AND d.COLLATION = c.COLLATION_NAME WHERE o.OBJECT_TYPE = 'DATABASE' 2025-10-27 11:28:56.022 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:56.022 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:56.022 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:56.022 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:56.023 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:56.026 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:28:56.173 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:56.179 INFO 2458 --- [refresh-tenant-resource-cache-6,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:28:56.268 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:56.274 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW GLOBAL VARIABLES WHERE Variable_name in ('lower_case_table_names') 2025-10-27 11:28:56.291 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:28:56.296 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: DROP DATABASE IF EXISTS `binlog_database`, args: [] 2025-10-27 11:28:56.622 WARN 2458 --- [ocp-async-6,2cd81f9e056c8c1a,79ca4ed6448181e7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:56.622 INFO 2458 --- [ocp-async-5,44366f8096cf23d9,66d07105ff1dc9ff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:56.631 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,0180e1b2c4fa09a5,f3f64714bd0fd8fe] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=0180e1b2c4fa09a5, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:56.669 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,0180e1b2c4fa09a5,f3f64714bd0fd8fe] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=0180e1b2c4fa09a5, duration=52 ms] 2025-10-27 11:28:56.707 INFO 2458 --- [prometheus-minute-schedule-1,02e3fbf65a99831b,b05ad088f5936c34] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:56.716 INFO 2458 --- [prometheus-minute-schedule-1,02e3fbf65a99831b,b05ad088f5936c34] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:56.868 INFO 2458 --- [ocp-schedules-13,b75763b9fd71fe55,d804177b876dd719] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:28:57.298 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] slow query, durationMillis=1002, sql=DROP DATABASE IF EXISTS `binlog_database` 2025-10-27 11:28:57.304 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.b.i.BinlogClusterMetadbManager : delete binlog metadb success, tenantId=2, database=binlog_database 2025-10-27 11:28:57.319 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9168, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:57.322 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,030e5f9c842ade67] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9172] 2025-10-27 11:28:57.465 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:28:57.467 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:28:57.561 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,a8ba508fcfedeada,af645d2dfb27afb2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=a8ba508fcfedeada, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:57.577 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,a8ba508fcfedeada,af645d2dfb27afb2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=a8ba508fcfedeada, duration=26 ms] 2025-10-27 11:28:57.747 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3aee037b6473e696,f41ae42dac2c6922] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=3aee037b6473e696, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:57.767 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,3aee037b6473e696,f41ae42dac2c6922] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=3aee037b6473e696, duration=30 ms] 2025-10-27 11:28:57.776 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,0db481ee204da42e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9172, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:57.799 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:57.805 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9172, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:57.774+08:00, sub_task_instance_name=Delete binlog cluster info, sub_task_instance_id=9172}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:57.873 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] c.o.o.c.s.o.OrganizationResourceManager : delete resource for current organization,organizationId=10000000,resourceType=4, resourceId=4 2025-10-27 11:28:57.900 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] c.o.o.b.i.t.DeleteBinlogClusterInfoTask : delete binlog cluster info done, binlogClusterId=4 2025-10-27 11:28:57.913 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9172, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:57.916 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,eb56d5be17a4b6fc] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9167] 2025-10-27 11:28:58.536 INFO 2458 --- [ocp-async-5,7c8918b6ac88c898,4257ad54d271d6f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:28:58.622 INFO 2458 --- [ocp-async-4,9f592f2161c0b67f,9d440a1eaa9a58c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:28:58.622 WARN 2458 --- [ocp-async-9,b547d93adaa1a178,94cc784c76c880f2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:28:58.807 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,0079d9720bb7f040] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9167, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:58.842 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3f6f6624ddd84640,d98e09df1a583659] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=3f6f6624ddd84640, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:58.883 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,3f6f6624ddd84640,d98e09df1a583659] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=3f6f6624ddd84640, duration=50 ms] 2025-10-27 11:28:58.878 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:58.885 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9167, context=Context{parallelIdx=-1, stringMap={subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:58.790+08:00, sub_task_instance_name=Unreserve host, sub_task_instance_id=9167}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:58.939 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.s.t.b.host.UnReserveHostTask : Unreserve host 4 for service BINLOG_SERVICE binlogtest2 2025-10-27 11:28:58.962 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.c.h.m.HostServiceManagerImpl : Removing service begin: service [BINLOG_SERVICE:binlogtest2], host [4:10.40.65.151] 2025-10-27 11:28:58.969 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.c.h.m.HostServiceManagerImpl : Removing service: no services left on host [4:10.40.65.151], status changed from [ONLINE] to [AVAILABLE] 2025-10-27 11:28:58.998 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.c.h.m.HostServiceManagerImpl : Removing service done: service [BINLOG_SERVICE:binlogtest2], host [4:10.40.65.151], current status [AVAILABLE] 2025-10-27 11:28:59.018 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9167, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:28:59.034 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8b7d929639189f7c] o.o.c.t.e.c.w.s.SubtaskFinishedProcessor : Set subtask to ready, subtaskIds=[9171] 2025-10-27 11:28:59.101 INFO 2458 --- [ocp-schedules-21,2372e9b248d8f5ed,579bd4daef073a1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:28:59.143 WARN 2458 --- [scanner-112195401-1,2372e9b248d8f5ed,faf90d4719065a97] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:28:59.338 INFO 2458 --- [alarm-detect-20,2372e9b248d8f5ed,319d96702736e0ae] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535710 ~ 1761535729 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535725, value=115.106)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535725, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535725=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:28:59.382 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5999d92b3909126d,52063fa952fd1f28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=5999d92b3909126d, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:28:59.382 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,5999d92b3909126d,52063fa952fd1f28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=5999d92b3909126d, duration=1 ms] 2025-10-27 11:28:59.468 INFO 2458 --- [ocp-schedules-7,965f64f7bc8f5463,9e987e042c262049] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:28:59.578 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0978e64222a67f43,f6f89b3ce2b38644] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=0978e64222a67f43, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:28:59.598 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,0978e64222a67f43,f6f89b3ce2b38644] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=0978e64222a67f43, duration=32 ms] 2025-10-27 11:28:59.707 INFO 2458 --- [prometheus-minute-schedule-1,7050797f2db93ddb,c02258e2b5fac53f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535740, interval=60 2025-10-27 11:28:59.724 INFO 2458 --- [prometheus-minute-schedule-1,7050797f2db93ddb,c02258e2b5fac53f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:28:59.823 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,fbada6fab2cfa4eb] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9171, operation:EXECUTE, state: RUNNING 2025-10-27 11:28:59.847 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:28:59.853 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9171, context=Context{parallelIdx=0, stringMap={4BINLOG_SERVICE=7, subtask_splitter=host_ids, service_name=binlogtest2, install_path=/home/admin/binlogservice, task_instance_id=9107, task_operation=execute, service_version=4.0.1-100000162024122511, binlog_cluster_id=4, service_type=BINLOG_SERVICE, ocpagent_service_name=logproxy_agent, operation_scope=Cluster, latest_execution_start_time=2025-10-27T11:28:59.820+08:00, sub_task_instance_name=Inactive agent modules, sub_task_instance_id=9171}, listMap={binlog_server_ids=[4], host_ids=[4], binlog_instance_names=[]}}, executor=10.40.65.146 2025-10-27 11:28:59.858 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.s.t.b.o.InactiveAgentModuleTask : inactive agent modules begin 2025-10-27 11:28:59.875 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:28:59.885 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:28:59.952 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,25ca0fdcd14ffa95,1456840c1da52bd9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=25ca0fdcd14ffa95, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:28:59.977 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,25ca0fdcd14ffa95,1456840c1da52bd9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=25ca0fdcd14ffa95, duration=33 ms] 2025-10-27 11:29:00.048 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : get hostServiceType LOG_PROXY, exporters {http://10.40.65.151:62889/metrics/logproxy=1} 2025-10-27 11:29:00.081 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:00.089 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:00.197 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:29:00.205 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : Inactiving OCP monitor module, host=4, mode=BINLOG_SERVICE_DEPLOYED 2025-10-27 11:29:00.225 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:00.234 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:00.302 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(monagent.pipeline.logproxy.status=inactive)]), params:null 2025-10-27 11:29:00.424 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:00.435 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:00.467 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:00.472 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:00.502 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.agent.HostAgentServiceImpl : get hostServiceType LOG_PROXY, exporters {http://10.40.65.151:62889/metrics/logproxy=1} 2025-10-27 11:29:00.509 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.s.t.b.o.InactiveAgentModuleTask : inactive agent modules end 2025-10-27 11:29:00.528 INFO 2458 --- [manual-subtask-executor16,90c99b6302573800,8a4bf30ca4625ebe] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9171, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:29:00.622 WARN 2458 --- [ocp-async-1,c3db57a93b4f5c6f,7f0dbaa61fade96c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:00.622 INFO 2458 --- [ocp-async-10,aa0d88cf42c03936,ff9e82c2fd72cc26] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:01.058 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6ec0714082e13d66,73e6729a277fe50f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=6ec0714082e13d66, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:29:01.086 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6ec0714082e13d66,73e6729a277fe50f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=6ec0714082e13d66, duration=37 ms] 2025-10-27 11:29:01.558 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,02abf7be61f4691a,0a2b4131d5977c44] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=02abf7be61f4691a, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:29:01.578 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,02abf7be61f4691a,0a2b4131d5977c44] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=02abf7be61f4691a, duration=30 ms] 2025-10-27 11:29:01.690 WARN 2458 --- [task-coordinator3,,] c.o.o.alarm.service.OcpAlarmServiceImpl : alarm type binlog_cluster_operation_info is not enabled 2025-10-27 11:29:01.692 INFO 2458 --- [task-coordinator3,,] c.o.o.s.operation.OperationAlarmService : create operation alarm, alarm={"status":"Disabled"} 2025-10-27 11:29:01.692 INFO 2458 --- [task-coordinator3,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9107, status:SUCCESS 2025-10-27 11:29:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:01.715283+08:00, lastProcessTime=2025-10-27T11:28:51.721732+08:00 2025-10-27 11:29:01.967 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:01.715283+08:00 2025-10-27 11:29:02.162 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a088b69e4d599a31,8a0ce9eeff8cc068] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/9107?taskInstanceId=9107, client=10.40.198.13, traceId=a088b69e4d599a31, method=SuccessResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskInstance(Long), args=9107,] 2025-10-27 11:29:02.192 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a088b69e4d599a31,8a0ce9eeff8cc068] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/9107 client=10.40.198.13, traceId=a088b69e4d599a31, duration=41 ms] 2025-10-27 11:29:02.622 WARN 2458 --- [ocp-async-8,ada955052a48f38d,2a422388e18da3cf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:02.622 INFO 2458 --- [ocp-async-5,c79133fd3aa262f9,fe0b885b9fc7ec4c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:02.707 INFO 2458 --- [prometheus-minute-schedule-1,92a48adc6cbbdc7e,297fd45b40b8590c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:02.717 INFO 2458 --- [prometheus-minute-schedule-1,92a48adc6cbbdc7e,297fd45b40b8590c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:03.473 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:03.476 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:03.568 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,244e078c0050a2b7,95e22feac7104fe5] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances?binlogClusterId=4&status=RUNNING%2CFAILED, client=10.40.198.13, traceId=244e078c0050a2b7, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listTaskInstances(String,Pageable,String,List,String,TaskType,String,Long,String,Long,Long,Long,Long,Long), args=Page request [number: 0, size 5000, sort: id: DESC],RUNNING,FAILED,MANUAL,4,] 2025-10-27 11:29:03.577 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,244e078c0050a2b7,95e22feac7104fe5] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances client=10.40.198.13, traceId=244e078c0050a2b7, duration=23 ms] 2025-10-27 11:29:04.208 INFO 2458 --- [ocp-schedules-25,9e7f660d1b6670bf,26cd27ab167d9fb6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:29:04.214 INFO 2458 --- [ocp-schedules-25,9e7f660d1b6670bf,26cd27ab167d9fb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:04.387 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:29:04.388 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:04.389 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:29:04.401 INFO 2458 --- [ocp-schedules-25,9e7f660d1b6670bf,26cd27ab167d9fb6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:29:04.622 WARN 2458 --- [ocp-async-9,bb8ec22645990526,b16ef414b9132463] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:04.622 INFO 2458 --- [ocp-async-4,12dfcf634e3d31ec,8940bf572934f1e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:05.264 INFO 2458 --- [ocp-schedules-21,b40369f8f7a8fb86,6e63a2e2de067692] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:05.268 INFO 2458 --- [ocp-schedules-21,b40369f8f7a8fb86,6e63a2e2de067692] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:05.707 INFO 2458 --- [prometheus-minute-schedule-1,0ecb3713c35e922c,9bd9c9da9c933c33] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:05.722 INFO 2458 --- [prometheus-minute-schedule-1,0ecb3713c35e922c,9bd9c9da9c933c33] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:06.476 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:06.477 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:06.622 WARN 2458 --- [ocp-async-7,8fab26d8f3fc3ef1,ac8af7d8980bb903] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:06.622 INFO 2458 --- [ocp-async-7,6a32af3def73f00c,c27687c631133dea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:06.874 INFO 2458 --- [ocp-schedules-28,463fdb72aa7bc5d6,f050b42e96b728b7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:07.531 INFO 2458 --- [ocp-schedules-14,24dadd793f5136a0,5edd2a627b0218ab] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:29:07.545 INFO 2458 --- [ocp-schedules-14,24dadd793f5136a0,5edd2a627b0218ab] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:07.545 INFO 2458 --- [ocp-schedules-14,24dadd793f5136a0,5edd2a627b0218ab] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:08.536 INFO 2458 --- [ocp-async-7,880e130062d40e78,312bd08cb2e4975d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:08.623 WARN 2458 --- [ocp-async-6,cf0ea85fc57c6e1a,8614535cdb1d9c3d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:08.623 INFO 2458 --- [ocp-async-3,8ba92f314356f5c1,319455a0701a23c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:08.707 INFO 2458 --- [prometheus-minute-schedule-1,5d09b897d86d591f,8a2940878cc4aa06] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:08.714 INFO 2458 --- [prometheus-minute-schedule-1,5d09b897d86d591f,8a2940878cc4aa06] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:09.344 INFO 2458 --- [ocp-schedules-14,b54a9b6397d758ec,3fb438cc6d162528] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:29:09.382 WARN 2458 --- [scanner-112195695-1,b54a9b6397d758ec,11b8395db3bcbe5e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:29:09.477 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:09.478 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:10.047 INFO 2458 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:29:10.047 INFO 2458 --- [ocp-schedules-7,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:29:10.122 INFO 2458 --- [Check obproxy1,e05cedfcd5d884be,4e736b1174755bd7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:29:10.122 INFO 2458 --- [Collect obproxy1,bf42436af73e6959,8b577133723feac1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:29:10.122 INFO 2458 --- [Check obproxy1,e05cedfcd5d884be,4e736b1174755bd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:29:10.122 INFO 2458 --- [Collect obproxy1,bf42436af73e6959,8b577133723feac1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:29:10.151 INFO 2458 --- [Check obproxy1,e05cedfcd5d884be,4e736b1174755bd7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:29:10.153 INFO 2458 --- [Collect obproxy1,bf42436af73e6959,8b577133723feac1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:29:10.153 INFO 2458 --- [Check obproxy1,e05cedfcd5d884be,4e736b1174755bd7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:29:10.153 INFO 2458 --- [Check obproxy1,e05cedfcd5d884be,4e736b1174755bd7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:29:10.157 INFO 2458 --- [Collect obproxy1,bf42436af73e6959,8b577133723feac1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:29:10.160 INFO 2458 --- [Collect obproxy1,bf42436af73e6959,8b577133723feac1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:29:10.199 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:29:10.623 WARN 2458 --- [ocp-async-9,7924492cf448d368,053d7cb60bcaa54f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:10.623 INFO 2458 --- [ocp-async-4,ed558abb6320ec55,b66af14bbb547505] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:11.410 INFO 2458 --- [ocp-schedules-28,f54326b35945c662,1d355e96c3d1624d] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:29:11.417 INFO 2458 --- [ocp-schedules-28,f54326b35945c662,1d355e96c3d1624d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:11.568 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:29:11.569 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.572 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:29:11.581 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.581 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.584 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=1000, affectRows=1000 2025-10-27 11:29:11.588 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=994, validCount=992 2025-10-27 11:29:11.607 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.608 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.633 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.634 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.636 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.636 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.664 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.664 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.676 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=1000, affectRows=1000 2025-10-27 11:29:11.681 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=999, validCount=995 2025-10-27 11:29:11.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:29:11.685 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.689 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:29:11.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.692 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.696 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587947) Duplicate entry '835370602979598336-1761535620' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587947) Duplicate entry '835370602979598336-1761535620' for key 'PRIMARY' 2025-10-27 11:29:11.704 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.705 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.707 INFO 2458 --- [prometheus-minute-schedule-1,8c79fc334b8f42d4,c556b1e74c4cb77b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:11.714476+08:00, lastProcessTime=2025-10-27T11:29:01.715283+08:00 2025-10-27 11:29:11.720 INFO 2458 --- [prometheus-minute-schedule-1,8c79fc334b8f42d4,c556b1e74c4cb77b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:11.755 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.757 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:11.758 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.759 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:11.760 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.761 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:11.761 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.762 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:11.762 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.763 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:11.764 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.764 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:11.765 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:29:11.766 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.767 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.769 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:29:11.792 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.793 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:29:11.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.831 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:11.831 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.831 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.831 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.832 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:29:11.841 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:11.841 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:29:11.841 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:11.841 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:29:11.844 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:29:11.844 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:29:11.862 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.863 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:29:11.865 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.866 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:29:11.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:29:11.937 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.938 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:29:11.940 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.941 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:29:11.967 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.968 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:29:11.978 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:11.714476+08:00 2025-10-27 11:29:11.997 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:11.997 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:29:12.022 INFO 2458 --- [metric-archive-async-2,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535680, nodeCount=821, affectRows=821 2025-10-27 11:29:12.052 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.053 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:12.053 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.054 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:12.055 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.055 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:12.056 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.057 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:12.057 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.058 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:12.059 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.059 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:12.060 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.061 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:12.061 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.063 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:12.063 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.064 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:12.065 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.066 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:29:12.066 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.067 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:29:12.067 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.068 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:29:12.069 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:29:12.071 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:29:12.168 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:12.168 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:12.169 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:29:12.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:29:12.202 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:29:12.202 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:29:12.203 INFO 2458 --- [ocp-schedules-28,f54326b35945c662,1d355e96c3d1624d] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:29:12.478 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:12.480 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:12.623 WARN 2458 --- [ocp-async-10,97ef8a59842d731e,7238fd022ddf3ba9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:12.623 INFO 2458 --- [ocp-async-10,5b951b5a039928ed,b5ba4cfc428e2ab7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:12.973 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=213, affectRows=213 2025-10-27 11:29:12.974 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=212, validCount=210 2025-10-27 11:29:13.001 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=979, affectRows=979 2025-10-27 11:29:13.005 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=974, validCount=972 2025-10-27 11:29:13.019 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=907, affectRows=907 2025-10-27 11:29:13.023 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=903, validCount=899 2025-10-27 11:29:14.623 WARN 2458 --- [ocp-async-8,a5c5dcbd947daaef,d4ab8d898ec07bad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:14.623 INFO 2458 --- [ocp-async-5,bd907588f1c4774c,63d7b3542de0602c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:14.707 INFO 2458 --- [prometheus-minute-schedule-1,1f1256013bf92d80,c5e122fef2776439] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:14.715 INFO 2458 --- [prometheus-minute-schedule-1,1f1256013bf92d80,c5e122fef2776439] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:15.276 INFO 2458 --- [ocp-schedules-5,7b2dea6741478117,8a9649743db91dbe] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:15.280 INFO 2458 --- [ocp-schedules-5,7b2dea6741478117,8a9649743db91dbe] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:15.481 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:15.482 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:15.856 INFO 2458 --- [ocp-schedules-1,002ebe4820830307,f71b5f3444ec0410] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:29:15.856+08:00, clusters=[1, 2] 2025-10-27 11:29:15.860 INFO 2458 --- [ocp-schedules-1,002ebe4820830307,f71b5f3444ec0410] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:29:15.861 INFO 2458 --- [ocp-schedules-1,002ebe4820830307,f71b5f3444ec0410] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:29:16.141 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535680, nodeCount=1000, affectRows=1000 2025-10-27 11:29:16.146 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535680, nodeCount=1000, validCount=987 2025-10-27 11:29:16.254 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,b0cf00760ba71f99,37ab8de5bed5bc64] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b0cf00760ba71f99, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:29:16.254 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,b0cf00760ba71f99,37ab8de5bed5bc64] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b0cf00760ba71f99, duration=1 ms] 2025-10-27 11:29:16.623 WARN 2458 --- [ocp-async-3,be55fce8f28ff619,64188f8e592d5f81] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:16.623 INFO 2458 --- [ocp-async-9,5294c26096d59b3f,021268caf5915b84] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:16.881 INFO 2458 --- [ocp-schedules-5,2742441b6540af14,c7e0c91cc6e173ff] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:17.567 INFO 2458 --- [ocp-schedules-19,62feb1e52aeaa1a9,2bba1eee70ea0334] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:29:17.567 INFO 2458 --- [ocp-schedules-19,62feb1e52aeaa1a9,2bba1eee70ea0334] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:29:17.623 INFO 2458 --- [Collect all tenants major compactions1,62feb1e52aeaa1a9,33c16a8a0ede8a48] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:29:17.623 INFO 2458 --- [Collect all tenants major compactions2,62feb1e52aeaa1a9,731d07f528e6fcfc] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:29:17.623 INFO 2458 --- [Collect all tenants major compactions3,62feb1e52aeaa1a9,d40eed4d602910aa] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:29:17.624 INFO 2458 --- [Collect all tenants major compactions4,62feb1e52aeaa1a9,b69f1ea1bb6821d8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:29:17.624 INFO 2458 --- [Collect all tenants major compactions6,62feb1e52aeaa1a9,e6b41ffc70969dee] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:29:17.624 INFO 2458 --- [Collect all tenants major compactions5,62feb1e52aeaa1a9,1931ef6e9999b7f9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:29:17.665 INFO 2458 --- [Collect all tenants major compactions5,62feb1e52aeaa1a9,1931ef6e9999b7f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.665 INFO 2458 --- [Collect all tenants major compactions6,62feb1e52aeaa1a9,e6b41ffc70969dee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.666 INFO 2458 --- [Collect all tenants major compactions5,62feb1e52aeaa1a9,1931ef6e9999b7f9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:29:17.667 INFO 2458 --- [Collect all tenants major compactions6,62feb1e52aeaa1a9,e6b41ffc70969dee] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:29:17.684 INFO 2458 --- [Collect all tenants major compactions6,62feb1e52aeaa1a9,e6b41ffc70969dee] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:29:17.685 INFO 2458 --- [Collect all tenants major compactions5,62feb1e52aeaa1a9,1931ef6e9999b7f9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:29:17.689 INFO 2458 --- [Collect all tenants major compactions1,62feb1e52aeaa1a9,33c16a8a0ede8a48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.689 INFO 2458 --- [Collect all tenants major compactions2,62feb1e52aeaa1a9,731d07f528e6fcfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.690 INFO 2458 --- [Collect all tenants major compactions1,62feb1e52aeaa1a9,33c16a8a0ede8a48] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:29:17.691 INFO 2458 --- [Collect all tenants major compactions2,62feb1e52aeaa1a9,731d07f528e6fcfc] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:29:17.691 INFO 2458 --- [Collect all tenants major compactions3,62feb1e52aeaa1a9,d40eed4d602910aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.693 INFO 2458 --- [Collect all tenants major compactions3,62feb1e52aeaa1a9,d40eed4d602910aa] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:29:17.694 INFO 2458 --- [Collect all tenants major compactions1,62feb1e52aeaa1a9,33c16a8a0ede8a48] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:29:17.694 INFO 2458 --- [Collect all tenants major compactions2,62feb1e52aeaa1a9,731d07f528e6fcfc] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:29:17.695 INFO 2458 --- [Collect all tenants major compactions4,62feb1e52aeaa1a9,b69f1ea1bb6821d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:17.696 INFO 2458 --- [Collect all tenants major compactions4,62feb1e52aeaa1a9,b69f1ea1bb6821d8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:29:17.697 INFO 2458 --- [Collect all tenants major compactions3,62feb1e52aeaa1a9,d40eed4d602910aa] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:29:17.699 INFO 2458 --- [Collect all tenants major compactions4,62feb1e52aeaa1a9,b69f1ea1bb6821d8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:29:17.700 INFO 2458 --- [ocp-schedules-19,62feb1e52aeaa1a9,2bba1eee70ea0334] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:29:17.707 INFO 2458 --- [prometheus-minute-schedule-1,13930f6862921645,99b3feffc8ce3d2e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:17.713 INFO 2458 --- [prometheus-minute-schedule-1,13930f6862921645,99b3feffc8ce3d2e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:18.482 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:18.483 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:18.536 INFO 2458 --- [ocp-async-9,112aaadc10303f0c,b8e58ca87dcc88af] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:18.623 WARN 2458 --- [ocp-async-4,75c7dd8c033426a3,78d9dbc85cdc1f1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:18.623 INFO 2458 --- [ocp-async-7,efa709898427de32,1749df4da82c687d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:19.461 INFO 2458 --- [ocp-schedules-14,51000f0c97dac920,cfe73fc552d56bb1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:29:19.502 WARN 2458 --- [scanner-112197393-1,51000f0c97dac920,e97a837a2110f51e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:29:19.659 INFO 2458 --- [alarm-detect-8,51000f0c97dac920,96069273330baafb] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535725 ~ 1761535749 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535740, value=60.347)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535740, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535740=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:29:19.662 INFO 2458 --- [alarm-detect-8,51000f0c97dac920,96069273330baafb] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:29:20.418 INFO 2458 --- [ocp-schedules-22,4cc37e8a557fede9,4ceabe46b8fec754] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:29:20.623 WARN 2458 --- [ocp-async-7,24fd46f65f115b40,22e634c6ccf0b263] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:20.623 INFO 2458 --- [ocp-async-8,5115baaf9ac70237,99605262172a2b0d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:20.707 INFO 2458 --- [prometheus-minute-schedule-1,8f53f3e4d7bd0fe1,5a8d50f0ec3bff76] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:20.714 INFO 2458 --- [prometheus-minute-schedule-1,8f53f3e4d7bd0fe1,5a8d50f0ec3bff76] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:21.442 INFO 2458 --- [ocp-schedules-7,ddb837409256f97f,04f5e34086af7de6] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:29:21.443 INFO 2458 --- [ocp-schedules-23,cf03deef3ea69f1f,9f19d6dc3bb1e417] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:21.443 INFO 2458 --- [ocp-schedules-4,35f754b1f5243f84,5279d150c17e1d20] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:21.443 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:29:21.443 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:29:21.444 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:29:21.444 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:29:21.445 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:29:21.445 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:29:21.445 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:29:21.445 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:29:21.445 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:29:21.446 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:29:21.447 INFO 2458 --- [ocp-schedules-1,c9f51f10d5787848,d2e076142e44b16c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:21.447 INFO 2458 --- [host-check-clockdiff1,c9f51f10d5787848,18935bc251bdba53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:29:21.447 INFO 2458 --- [host-check-clockdiff2,c9f51f10d5787848,45e3396a95f2b2a1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:29:21.447 INFO 2458 --- [host-check-clockdiff3,c9f51f10d5787848,3e4aed4ae6b39a94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:29:21.448 INFO 2458 --- [host-check-clockdiff5,c9f51f10d5787848,159e2210ed57fe9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:29:21.451 INFO 2458 --- [host-check-clockdiff4,c9f51f10d5787848,4b990354460975ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:29:21.458 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:21.459 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:21.460 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:21.460 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:21.461 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:21.462 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:21.463 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:21.463 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:21.463 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:21.465 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:21.466 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:21.466 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:21.467 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:21.467 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:21.485 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:21.485 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:21.488 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:21.488 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:21.489 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:21.489 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:21.495 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:21.495 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:21.495 INFO 2458 --- [host-check-clockdiff1,c9f51f10d5787848,18935bc251bdba53] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535761 0 0 , error= 2025-10-27 11:29:21.502 INFO 2458 --- [host-check-clockdiff2,c9f51f10d5787848,45e3396a95f2b2a1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535761 -1 -1 , error= 2025-10-27 11:29:21.509 INFO 2458 --- [host-check-clockdiff4,c9f51f10d5787848,4b990354460975ad] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535761 0 0 , error= 2025-10-27 11:29:21.545 INFO 2458 --- [ocp-schedules-20,8bfdcf5b2d058107,4c8829f4f45e13c7] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:29:21.549 INFO 2458 --- [ocp-schedules-25,2e1c23623a613a5b,b1347759dbf8c556] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:29:21.553 INFO 2458 --- [ocp-schedules-20,8bfdcf5b2d058107,4c8829f4f45e13c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:21.554 INFO 2458 --- [ocp-schedules-6,9657349e55d61f8e,4d8a01afe07b1692] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:29:21.554 INFO 2458 --- [ocp-schedules-25,2e1c23623a613a5b,b1347759dbf8c556] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:21.559 INFO 2458 --- [ocp-schedules-6,9657349e55d61f8e,4d8a01afe07b1692] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:21.560 INFO 2458 --- [ocp-schedules-25,2e1c23623a613a5b,b1347759dbf8c556] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:29:21.566 INFO 2458 --- [ocp-schedules-22,f4e5b3881fdf8efa,255ba7de7feab484] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:29:21.716 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.717 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:21.718 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:29:21.727 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:29:21.731 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:29:21.731075643+08:00 2025-10-27 11:29:21.732 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:29:21.732 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:29:21.731+08:00 2025-10-27 11:29:21.732 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:21.731371+08:00, lastProcessTime=2025-10-27T11:29:11.714476+08:00 2025-10-27 11:29:21.734 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:29:21.734 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:29:21.734 INFO 2458 --- [host-check-agent-status4,35f754b1f5243f84,bc5d7155c69fe198] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:29:21.742 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:29:21.744 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:29:21.743557676+08:00 2025-10-27 11:29:21.745 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:29:21.744+08:00 2025-10-27 11:29:21.747 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:29:21.731+08:00 and status AVAILABLE 2025-10-27 11:29:21.749 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:29:21.749 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:29:21.750 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.751 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:21.751 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:29:21.751 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:29:21.752 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:29:21.752 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:29:21.752 INFO 2458 --- [host-check-agent-status3,35f754b1f5243f84,0a0dfb7dba64e662] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:29:21.753 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:29:21.751759717+08:00 2025-10-27 11:29:21.753 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:29:21.753 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:29:21.753 INFO 2458 --- [host-check-agent-status1,35f754b1f5243f84,9436332061f2af6c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:29:21.756 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:29:21.756 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:29:21.753+08:00 2025-10-27 11:29:21.758 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:29:21.758 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:29:21.758 INFO 2458 --- [host-check-agent-status2,35f754b1f5243f84,57c15abc9cdd094c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:29:21.761 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:29:21.744+08:00 and status AVAILABLE 2025-10-27 11:29:21.763 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:29:21.763 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:29:21.753+08:00 and status AVAILABLE 2025-10-27 11:29:21.764 INFO 2458 --- [ocp-schedules-20,8bfdcf5b2d058107,4c8829f4f45e13c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:29:21.766 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:29:21.768 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:29:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:21.788 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:21.790 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:21.791 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:21.791 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:21.792 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:21.794 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:21.807 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:29:21.807 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:29:21.807 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:29:21.807545184+08:00 2025-10-27 11:29:21.808 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:29:21.807+08:00 2025-10-27 11:29:21.809 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:29:21.809010040+08:00 2025-10-27 11:29:21.809 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:29:21.809+08:00 2025-10-27 11:29:21.811 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.811 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:21.824 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:29:21.827 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:29:21.809+08:00 and status AVAILABLE 2025-10-27 11:29:21.828 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:29:21.828 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:29:21.828 INFO 2458 --- [host-check-agent-status5,35f754b1f5243f84,9cd3e52ce328a4d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:29:21.830 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:29:21.807+08:00 and status AVAILABLE 2025-10-27 11:29:21.832 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:29:21.835 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:29:21.860 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:21.862 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:21.863 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:21.864 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:21.866 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:21.867 INFO 2458 --- [ocp-schedules-30,9767a8cce6af5ee7,b4ee3ba650526450] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:29:21.867 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:21.869 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:21.884 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:29:21.885 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:29:21.886 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:29:21.913 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.914 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:29:21.923 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:29:21.929 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:29:21.929 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:29:21.935 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:21.936 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:21.937 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:21.937 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.937 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:29:21.954 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:29:21.967 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:29:21.982 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:21.987 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:21.994 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:29:21.996 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:21.996 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:21.997 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:21.998 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.001 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:29:22.002 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:29:22.003 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:22.003 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:29:22.017 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:22.018 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:29:22.019 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:21.731371+08:00 2025-10-27 11:29:22.026 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:22.034 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:29:22.037 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:29:22.039 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:29:22.044 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:29:22.044 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:22.049 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.049 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:22.051 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.052 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:29:22.056 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:22.071 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:22.073 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:29:22.073 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:22.077 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.080 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:29:22.085 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:29:22.086 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:29:22.094 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:29:22.102 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:22.106 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.106 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:22.111 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:22.112 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.113 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:22.114 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:29:22.116 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:29:22.117 INFO 2458 --- [host-check-host-status4,cf03deef3ea69f1f,c73609f194266e50] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:29:22.120 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:22.122 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:29:22.123 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:29:22.132 INFO 2458 --- [host-check-host-status2,cf03deef3ea69f1f,7af3bb50cf3d1b1e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:29:22.134 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:22.137 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:22.140 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:29:22.147 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.147 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:22.149 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:22.150 INFO 2458 --- [host-check-host-status3,cf03deef3ea69f1f,eede60c0aa26293c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:29:22.170 INFO 2458 --- [host-check-host-status5,cf03deef3ea69f1f,09f56b542adfdf63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:29:22.178 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:29:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:22.191 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:29:22.204 INFO 2458 --- [host-check-host-status1,cf03deef3ea69f1f,32f5d4e7c0a3502c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:29:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:22.233 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:29:22.233 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.234 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:29:22.236 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.237 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:29:22.252 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:29:22.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:22.274 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:22.275 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.275 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:29:22.298 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:29:22.298 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:29:22.303 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:22.342 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:22.342 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.343 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:22.347 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:29:22.357 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:29:22.366 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:22.369 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:22.397 INFO 2458 --- [host-check-clockdiff5,c9f51f10d5787848,159e2210ed57fe9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535762 0 0 , error= 2025-10-27 11:29:22.427 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:29:22.502 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:29:22.503 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.504 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:29:22.506 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.507 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:29:22.543 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:29:22.584 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:22.584 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:22.585 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:29:22.624 WARN 2458 --- [ocp-async-8,d27fed2bac8615fc,114bc857b8c2baa3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:22.625 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:29:22.625 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:29:22.625 INFO 2458 --- [ocp-async-2,48da4059efab5dd6,a372e7ef6c2c3da8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:22.625 INFO 2458 --- [ocp-schedules-6,9657349e55d61f8e,4d8a01afe07b1692] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:29:23.243 INFO 2458 --- [host-check-clockdiff3,c9f51f10d5787848,3e4aed4ae6b39a94] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535763 0 0 , error= 2025-10-27 11:29:23.707 INFO 2458 --- [prometheus-minute-schedule-1,e83fd18b9097eb89,b7c49d9fa04f5b8b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:23.714 INFO 2458 --- [prometheus-minute-schedule-1,e83fd18b9097eb89,b7c49d9fa04f5b8b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:24.489 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:24.490 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:24.624 WARN 2458 --- [ocp-async-2,984d17bce8da8111,547aa8bb12d32a56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:24.624 INFO 2458 --- [ocp-async-1,a00e1009dc171b33,49ee24d504d2975f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:25.287 INFO 2458 --- [ocp-schedules-1,d2e11865264647fb,e0df8a8aefcba759] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:25.291 INFO 2458 --- [ocp-schedules-1,d2e11865264647fb,e0df8a8aefcba759] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:26.624 WARN 2458 --- [ocp-async-1,c49aacec896de88e,9fbb80e9fd029c55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:26.624 INFO 2458 --- [ocp-async-7,bc4d55445a15e58b,24dd654f84e457f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:26.707 INFO 2458 --- [prometheus-minute-schedule-1,f5b993a06bfcf421,e77799e3a74fd19e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:26.714 INFO 2458 --- [prometheus-minute-schedule-1,f5b993a06bfcf421,e77799e3a74fd19e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:26.887 INFO 2458 --- [ocp-schedules-7,fb00ec25695cde9a,fb6c3acd78943b9f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:27.490 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:27.492 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:27.550 INFO 2458 --- [ocp-schedules-11,c9862f66124c53c3,632225570a30dd47] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:29:27.559 INFO 2458 --- [ocp-schedules-11,c9862f66124c53c3,632225570a30dd47] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:27.559 INFO 2458 --- [ocp-schedules-11,c9862f66124c53c3,632225570a30dd47] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:28.536 INFO 2458 --- [ocp-async-6,2555caa3ce11f777,4f30624ed3fe6cf1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:28.624 WARN 2458 --- [ocp-async-3,33c10c5f882a92f8,00052dbabcdd8bb9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:28.624 INFO 2458 --- [ocp-async-9,74d92d6cf0b36e5d,356edf7560d24eb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:29.021 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:29:29.021 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:29:29.023 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:29:29.023 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:29:29.024 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:29:29.024 INFO 2458 --- [ocp-schedules-9,4c2e0232aab45910,d8786545fe33698f] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:29:29.470 INFO 2458 --- [ocp-schedules-29,8f8c19a61679a501,4869fb68089095c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:29:29.483 INFO 2458 --- [ocp-schedules-29,8f8c19a61679a501,4869fb68089095c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:29:29.483 INFO 2458 --- [ocp-schedules-29,8f8c19a61679a501,4869fb68089095c6] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:29:29.711 INFO 2458 --- [prometheus-minute-schedule-1,ea133c08fad9c354,4f9e0f8e874ce27a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:29.722 INFO 2458 --- [prometheus-minute-schedule-1,ea133c08fad9c354,4f9e0f8e874ce27a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:29.735 INFO 2458 --- [ocp-schedules-28,ec76825b30cad542,167552751b8d2552] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:29:29.752 WARN 2458 --- [scanner-112197612-1,ec76825b30cad542,9f87b5e1cea016bd] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:29:29.852 INFO 2458 --- [alarm-detect-2,ec76825b30cad542,2d116c76674a2c9c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535740 ~ 1761535759 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535755, value=57.775)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535755, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535755=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:29:30.492 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:30.493 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:30.624 WARN 2458 --- [ocp-async-9,6a02bd0e343711e1,3700972d9a51702b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:30.624 INFO 2458 --- [ocp-async-2,a2a01821b3111902,81dcae4fe910d0cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:31.716266+08:00, lastProcessTime=2025-10-27T11:29:21.731371+08:00 2025-10-27 11:29:31.916 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:31.716266+08:00 2025-10-27 11:29:32.624 WARN 2458 --- [ocp-async-2,2da3efe9d3337e2c,f7fb8e50e119f799] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:32.624 INFO 2458 --- [ocp-async-7,a3c1f4ab305aaa17,3666805f1b8d5270] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:32.707 INFO 2458 --- [prometheus-minute-schedule-1,1688e26f93c2c2e8,a0b03b5172bf86a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:32.714 INFO 2458 --- [prometheus-minute-schedule-1,1688e26f93c2c2e8,a0b03b5172bf86a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:33.493 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:33.494 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:33.940 INFO 2458 --- [ocp-schedules-7,c98487bb7315a1b5,ad191ae4adf6aa54] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:29:33.940 INFO 2458 --- [ocp-schedules-7,c98487bb7315a1b5,ad191ae4adf6aa54] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:29:34.624 WARN 2458 --- [ocp-async-8,0b9e8865aec7e7be,f44e39e496234f91] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:34.625 INFO 2458 --- [ocp-async-3,dad59fad43f07c5b,7458908e7a6750a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:35.299 INFO 2458 --- [ocp-schedules-22,67ec989df9c7efa4,80b992ae8683fe2c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:35.303 INFO 2458 --- [ocp-schedules-22,67ec989df9c7efa4,80b992ae8683fe2c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:35.707 INFO 2458 --- [prometheus-minute-schedule-1,edd5789e63bc43d3,7a5f5df1d6c386c4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:35.714 INFO 2458 --- [prometheus-minute-schedule-1,edd5789e63bc43d3,7a5f5df1d6c386c4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:36.494 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:36.495 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:36.625 INFO 2458 --- [ocp-async-4,3bb4ee8fbf911d11,339f611de9da3306] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:36.625 WARN 2458 --- [ocp-async-9,ba1f3f40500a75c9,abeffa7f32702fc4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:36.894 INFO 2458 --- [ocp-schedules-22,80b35bdfe9856e7f,746c4423c1954c60] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:37.164 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:29:37.164 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:29:37.168 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:29:37.168 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:29:37.168 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:29:37.168 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:29:37.172 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:29:37.172 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:29:37.172 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:29:37.172 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:29:37.196 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:29:37.201 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:29:37.201 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:29:37.201 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:29:37.201 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:29:37.201 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:29:37.207 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:29:37.634 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:29:37.708 INFO 2458 --- [prometheus-cache-monitor1,04d8c80ccc171b25,7e98d936234a82f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20390,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5074,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7888} 2025-10-27 11:29:37.708 INFO 2458 --- [prometheus-cache-monitor1,04d8c80ccc171b25,7e98d936234a82f6] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:29:38.030 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.857871206S 2025-10-27 11:29:38.030 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:29:38.096 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:29:38.096 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:29:38.136 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:29:38.186 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:29:38.194 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:29:38.318 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] o.s.boot.SpringApplication : Started application in 1.149 seconds (JVM running for 258246.744) 2025-10-27 11:29:38.318 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.149666622S 2025-10-27 11:29:38.537 INFO 2458 --- [ocp-async-4,53a2cab08e796948,1d55533c2d4215f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:38.625 INFO 2458 --- [ocp-async-2,f3f0e6c0fcfe469d,0d6b6cd284139c10] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:38.625 WARN 2458 --- [ocp-async-7,f6600fae4d768517,924af169282f859b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:38.707 INFO 2458 --- [ocp-schedules-25,f70bc6d902593671,6887afe1f332f6d3] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:29:38.707 INFO 2458 --- [prometheus-minute-schedule-1,f317fb4f4f0ae83d,42f52fa90c57f46e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:38.715 INFO 2458 --- [prometheus-minute-schedule-1,f317fb4f4f0ae83d,42f52fa90c57f46e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:38.803 INFO 2458 --- [ocp-schedules-10,1b9515063f787023,13825ba21f581cb4] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:29:38.810 INFO 2458 --- [ocp-schedules-10,1b9515063f787023,13825ba21f581cb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:38.810 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:29:38.810 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:29:38.810 INFO 2458 --- [ocp-schedules-10,1b9515063f787023,13825ba21f581cb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:29:39.349 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:29:39.349 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:29:39.495 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:39.496 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:39.497 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:29:39.506 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:39.510 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:39.511 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:29:39.537 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:39.537 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:29:39.537 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:29:39.538 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:29:39.538 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:29:39.538 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:29:39.565 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:39.566 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:39.568 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:39.569 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:39.570 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:39.572 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:39.572 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:39.573 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:39.575 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:39.576 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:39.585 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:39.586 INFO 2458 --- [ocp-schedules-5,e1e97305b31204ec,22db3369aa072f0e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:29:39.629 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:29:39.630 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:29:39.635 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:29:39.636 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:29:39.644 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:29:39.647 INFO 2458 --- [ocp-schedules-18,58fb40b8affd763e,6ebe252e72042482] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:29:39.866 INFO 2458 --- [ocp-schedules-30,d54752e03775c88a,5232e6ffdbd03172] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:29:39.909 WARN 2458 --- [scanner-112197839-1,d54752e03775c88a,3ba31bfc5fd2a93f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:29:40.018 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:29:40.018 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:29:40.051 INFO 2458 --- [Check obproxy1,647370cacb5a7844,68775cd41a0680b6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:29:40.055 INFO 2458 --- [Check obproxy1,647370cacb5a7844,68775cd41a0680b6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:29:40.061 INFO 2458 --- [Collect obproxy1,638c8c28746bc4fb,cc5cfef2b812c399] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:29:40.061 INFO 2458 --- [Collect obproxy1,638c8c28746bc4fb,cc5cfef2b812c399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:29:40.089 INFO 2458 --- [Check obproxy1,647370cacb5a7844,68775cd41a0680b6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:29:40.089 INFO 2458 --- [Collect obproxy1,638c8c28746bc4fb,cc5cfef2b812c399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:29:40.092 INFO 2458 --- [Check obproxy1,647370cacb5a7844,68775cd41a0680b6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:29:40.092 INFO 2458 --- [Check obproxy1,647370cacb5a7844,68775cd41a0680b6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:29:40.093 INFO 2458 --- [Collect obproxy1,638c8c28746bc4fb,cc5cfef2b812c399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:29:40.096 INFO 2458 --- [Collect obproxy1,638c8c28746bc4fb,cc5cfef2b812c399] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:29:40.625 INFO 2458 --- [ocp-async-8,0de37d04a76e086d,c3935a7010142cb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:40.625 WARN 2458 --- [ocp-async-6,b3074c3806abdf9d,eda3657d68348597] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:41.271 INFO 2458 --- [ocp-schedules-22,1eb0ea43d751caa7,206743eb09d5c8af] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:29:41.707 INFO 2458 --- [prometheus-minute-schedule-1,33348ecbf9f57a1d,5a4d421707d77e86] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:41.713 INFO 2458 --- [prometheus-minute-schedule-1,33348ecbf9f57a1d,5a4d421707d77e86] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:41.714413+08:00, lastProcessTime=2025-10-27T11:29:31.716266+08:00 2025-10-27 11:29:41.926 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:41.714413+08:00 2025-10-27 11:29:42.506 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:42.507 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:42.625 WARN 2458 --- [ocp-async-9,fc274cf53c3addf9,f457ec0a85d12492] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:42.625 INFO 2458 --- [ocp-async-9,ad88bf953818d731,a5068e198c1981f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:44.238 INFO 2458 --- [ocp-schedules-29,a3a198dbf1c97c2f,1900665f73c913e9] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:29:44.625 WARN 2458 --- [ocp-async-5,4b2cae1fa558d202,4cd6e09dfcb706c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:44.625 INFO 2458 --- [ocp-async-5,b5b6cd6c4c968a04,4adb30dc75d1a521] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:44.707 INFO 2458 --- [prometheus-minute-schedule-1,542788de8c79a254,070d357bc09c15ff] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:44.714 INFO 2458 --- [prometheus-minute-schedule-1,542788de8c79a254,070d357bc09c15ff] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:45.102 INFO 2458 --- [prometheus-second-schedule-1,595dc0363d5b2c2b,3d2c8c163fae3743] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:29:45.312 INFO 2458 --- [ocp-schedules-8,b2e0291b93970420,04525e225cbabdcd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:45.315 INFO 2458 --- [ocp-schedules-8,b2e0291b93970420,04525e225cbabdcd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:45.507 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:45.508 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:46.625 INFO 2458 --- [ocp-async-2,c8235e0995dbf882,ec084d12715cb24f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:46.625 WARN 2458 --- [ocp-async-6,da44febd06e67bc2,085780d6abba2e45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:46.773 INFO 2458 --- [ocp-schedules-2,5a27d769a2e473b7,f2c13234e2d0be60] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:29:46.773 INFO 2458 --- [ocp-schedules-2,5a27d769a2e473b7,f2c13234e2d0be60] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:29:46.825 INFO 2458 --- [pool-881-thread-2,5a27d769a2e473b7,f52ce6162fce2571] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:29:46.825 INFO 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:29:46.883 INFO 2458 --- [pool-881-thread-2,5a27d769a2e473b7,f52ce6162fce2571] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:46.884 INFO 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:46.886 INFO 2458 --- [pool-881-thread-2,5a27d769a2e473b7,f52ce6162fce2571] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535486000000, 1761535786000000] 2025-10-27 11:29:46.887 INFO 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535486000000, 1761535786000000] 2025-10-27 11:29:46.900 INFO 2458 --- [ocp-schedules-9,c7f32c39e34e544b,dd3b589f641f3d4e] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:46.931 WARN 2458 --- [pool-881-thread-2,5a27d769a2e473b7,f52ce6162fce2571] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:29:46.951 INFO 2458 --- [pool-881-thread-2,5a27d769a2e473b7,f52ce6162fce2571] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:29:46.992 WARN 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:29:46.992 WARN 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:29:46.992 WARN 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:29:47.001 INFO 2458 --- [pool-881-thread-1,5a27d769a2e473b7,9b04f33d1b31d77a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:29:47.002 INFO 2458 --- [ocp-schedules-2,5a27d769a2e473b7,f2c13234e2d0be60] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:29:47.562 INFO 2458 --- [ocp-schedules-19,13e1af4852847c28,a5f2c6c4b49eae58] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:29:47.573 INFO 2458 --- [ocp-schedules-19,13e1af4852847c28,a5f2c6c4b49eae58] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:47.573 INFO 2458 --- [ocp-schedules-19,13e1af4852847c28,a5f2c6c4b49eae58] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:29:47.707 INFO 2458 --- [prometheus-minute-schedule-1,e081d0c0d9493943,baf0bb5215da0fb3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:47.714 INFO 2458 --- [prometheus-minute-schedule-1,e081d0c0d9493943,baf0bb5215da0fb3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:48.508 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:48.509 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:48.537 INFO 2458 --- [ocp-async-2,1a4dcb1cf739b62c,26438fb89fdfd072] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:48.625 INFO 2458 --- [ocp-async-10,026e01b8ee0e46d3,2d6e75386118a9ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:48.625 WARN 2458 --- [ocp-async-9,c0330183c58c5da3,ce10a7c73d01ac11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:49.585 INFO 2458 --- [ocp-schedules-19,a6909261218222cc,0531be433ae1eac5] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:29:49.585+08:00, clusters=[1, 2] 2025-10-27 11:29:49.600 INFO 2458 --- [ocp-schedules-19,a6909261218222cc,0531be433ae1eac5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:29:49.602 INFO 2458 --- [ocp-schedules-19,a6909261218222cc,0531be433ae1eac5] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:29:50.014 INFO 2458 --- [ocp-schedules-27,86516c4f98903e88,fa6ff4d8d7222493] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:29:50.050 WARN 2458 --- [scanner-112198056-1,86516c4f98903e88,a9d325e6dadd7ff8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:29:50.157 INFO 2458 --- [alarm-detect-19,86516c4f98903e88,0c91fc9df2604358] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535755 ~ 1761535780 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535770, value=61.227)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535770, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535770=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:29:50.626 INFO 2458 --- [ocp-async-1,7a845e3feafd7310,900377d397475388] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:50.626 WARN 2458 --- [ocp-async-7,25231b249eb01c77,7db56d0c3afef33c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:50.707 INFO 2458 --- [prometheus-minute-schedule-1,362322950ea5269b,4b92fd1b2f14e38e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:50.714 INFO 2458 --- [prometheus-minute-schedule-1,362322950ea5269b,4b92fd1b2f14e38e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:51.431 INFO 2458 --- [ocp-schedules-30,f02c9fc692e478b8,6055b1977cbfe86c] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:51.431 INFO 2458 --- [ocp-schedules-16,0c6d85fd5ccfc65d,231fcf6ff707a1de] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:51.432 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:29:51.432 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:29:51.432 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:29:51.432 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:29:51.432 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:29:51.433 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:29:51.433 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:29:51.433 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:29:51.433 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:29:51.433 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:29:51.442 INFO 2458 --- [ocp-schedules-7,d76447756f5e8a37,57d1b3ba63def95b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:29:51.447 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:51.448 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:51.448 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:51.448 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:51.449 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:51.449 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:51.450 INFO 2458 --- [ocp-schedules-10,64e1c9277144fc3a,16dc11a3a6817458] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:29:51.450 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:51.450 INFO 2458 --- [host-check-clockdiff1,64e1c9277144fc3a,87a76240db05005a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:29:51.450 INFO 2458 --- [host-check-clockdiff2,64e1c9277144fc3a,5fc51fd50c33a2b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:29:51.450 INFO 2458 --- [host-check-clockdiff3,64e1c9277144fc3a,52e2df4b53b62648] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:29:51.451 INFO 2458 --- [host-check-clockdiff4,64e1c9277144fc3a,034c2adf96b24b8c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:29:51.451 INFO 2458 --- [host-check-clockdiff5,64e1c9277144fc3a,485dd31d18513524] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:29:51.453 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:51.453 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:51.453 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:51.453 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:51.453 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:51.454 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:51.454 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:51.454 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:51.457 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:51.457 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:51.458 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:51.474 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:51.475 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:51.490 INFO 2458 --- [host-check-clockdiff2,64e1c9277144fc3a,5fc51fd50c33a2b4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535791 -1 -1 , error= 2025-10-27 11:29:51.490 INFO 2458 --- [host-check-clockdiff4,64e1c9277144fc3a,034c2adf96b24b8c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535791 -1 -1 , error= 2025-10-27 11:29:51.490 INFO 2458 --- [host-check-clockdiff1,64e1c9277144fc3a,87a76240db05005a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535791 0 0 , error= 2025-10-27 11:29:51.490 INFO 2458 --- [host-check-clockdiff5,64e1c9277144fc3a,485dd31d18513524] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535791 0 0 , error= 2025-10-27 11:29:51.509 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:51.510 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:51.531 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:29:51.532 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:29:51.533 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:29:51.533203404+08:00 2025-10-27 11:29:51.534 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:29:51.534 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:29:51.533+08:00 2025-10-27 11:29:51.534 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:29:51.535 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:29:51.536 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:29:51.535357866+08:00 2025-10-27 11:29:51.536 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:29:51.535370686+08:00 2025-10-27 11:29:51.536 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:29:51.536 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:29:51.536 INFO 2458 --- [host-check-agent-status5,f02c9fc692e478b8,b916083efa4c3685] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:29:51.537 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:29:51.536+08:00 2025-10-27 11:29:51.537 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:29:51.536+08:00 2025-10-27 11:29:51.537 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:29:51.537 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:29:51.537 INFO 2458 --- [host-check-agent-status4,f02c9fc692e478b8,5471ba52fcfcf77f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:29:51.538 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:29:51.539 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:29:51.540 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:29:51.541 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:29:51.541 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:29:51.541 INFO 2458 --- [host-check-agent-status1,f02c9fc692e478b8,384822462f308a24] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status2,f02c9fc692e478b8,a9e5bdf969bfd98f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:29:51.543 INFO 2458 --- [host-check-agent-status3,f02c9fc692e478b8,c7eabb27371f2639] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:29:51.545 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:29:51.533+08:00 and status AVAILABLE 2025-10-27 11:29:51.545 INFO 2458 --- [ocp-schedules-18,2c1c8c4c8293fe78,87696182ab7ea235] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:29:51.546 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:29:51.546 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:29:51.546668108+08:00 2025-10-27 11:29:51.547 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:29:51.536+08:00 and status AVAILABLE 2025-10-27 11:29:51.547 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:29:51.547+08:00 2025-10-27 11:29:51.547 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:29:51.536+08:00 and status AVAILABLE 2025-10-27 11:29:51.549 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:29:51.549 INFO 2458 --- [ocp-schedules-1,6f5bed53886d661d,173d12fa97085607] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:29:51.549 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:29:51.548770232+08:00 2025-10-27 11:29:51.549 INFO 2458 --- [ocp-schedules-18,2c1c8c4c8293fe78,87696182ab7ea235] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:51.550 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:29:51.549+08:00 2025-10-27 11:29:51.550 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:29:51.552 INFO 2458 --- [ocp-schedules-1,6f5bed53886d661d,173d12fa97085607] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:51.553 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:29:51.553 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:29:51.553 INFO 2458 --- [ocp-schedules-1,6f5bed53886d661d,173d12fa97085607] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:29:51.554 INFO 2458 --- [ocp-schedules-12,398cd241a3b9f57f,bc25b6166412b7eb] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:29:51.555 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:29:51.547+08:00 and status AVAILABLE 2025-10-27 11:29:51.557 INFO 2458 --- [ocp-schedules-12,398cd241a3b9f57f,bc25b6166412b7eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:51.559 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:29:51.549+08:00 and status AVAILABLE 2025-10-27 11:29:51.561 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:29:51.563 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:29:51.565 INFO 2458 --- [ocp-schedules-13,4cb05b7bc77e5cbb,2b7030b4126fc36b] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:29:51.568 INFO 2458 --- [ocp-schedules-14,4303a4c149eca3c2,f759fab003ff03be] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:29:51.571 INFO 2458 --- [ocp-schedules-20,773b269a2b0f7b00,7395db875efb5b76] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:29:51.572 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:29:51.573 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:51.576 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:51.576 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:29:51.576 INFO 2458 --- [ocp-schedules-14,4303a4c149eca3c2,f759fab003ff03be] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:29:51.579 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:29:51.580 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:51.582 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:29:51.583 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:51.586 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.587 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:51.588 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:29:51.589 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:29:51.592 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:29:51.595 INFO 2458 --- [ocp-schedules-20,773b269a2b0f7b00,7395db875efb5b76] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:29:51.624 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.625 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:51.633 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:29:51.633 INFO 2458 --- [ocp-schedules-18,2c1c8c4c8293fe78,87696182ab7ea235] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:29:51.642 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:51.643 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:51.647 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:51.648 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:51.657 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:29:51.658 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:29:51.659 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:29:51.668 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:29:51.676 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.677 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:29:51.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:51.678 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:29:51.691 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:29:51.706 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:29:51.709 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:51.712 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:29:51.713 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.714 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:29:51.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:29:51.713053+08:00, lastProcessTime=2025-10-27T11:29:41.714413+08:00 2025-10-27 11:29:51.716 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:29:51.717 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:29:51.741 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:51.743 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:51.744 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:51.748 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:51.755 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:29:51.756 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:29:51.757 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:51.758 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.759 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:29:51.761 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:29:51.763 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:29:51.776 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:29:51.777 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:29:51.804 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:51.810 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.811 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:29:51.813 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:29:51.814 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:51.815 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:29:51.816 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:29:51.828 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:29:51.836 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:29:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:51.847 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:51.852 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:51.853 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:51.855 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:51.864 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:29:51.866 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:29:51.867 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:29:51.870 INFO 2458 --- [ocp-schedules-11,f7849b58584ff52b,98be8d8d8c9e23cc] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:29:51.871 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:51.873 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.874 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:29:51.875 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.876 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:29:51.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:29:51.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:51.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:51.882 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:29:51.899 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:29:51.900 INFO 2458 --- [host-check-host-status4,0c6d85fd5ccfc65d,3bbc80f46b84f604] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:29:51.903 INFO 2458 --- [host-check-host-status5,0c6d85fd5ccfc65d,b0c8fdde9c67e6a1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:29:51.906 INFO 2458 --- [host-check-host-status3,0c6d85fd5ccfc65d,7eac96fba72d72c5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:29:51.917 INFO 2458 --- [host-check-host-status2,0c6d85fd5ccfc65d,2b2f2f7ef60ba573] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:29:51.919 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:51.920 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:51.920 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:51.921 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:29:51.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:51.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:51.931 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:29:51.932 INFO 2458 --- [host-check-host-status1,0c6d85fd5ccfc65d,9350b25be4722044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:29:51.944 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:29:51.947 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:29:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:51.971 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:51.974 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:29:51.713053+08:00 2025-10-27 11:29:51.998 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:29:52.008 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:52.008 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:52.041 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:29:52.041 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.042 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:29:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:52.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:29:52.048 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:29:52.066 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:29:52.080 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:52.080 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:29:52.090 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:52.094 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.094 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:29:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:29:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:29:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:29:52.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:52.155 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.156 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:29:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:29:52.172 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:29:52.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:29:52.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:29:52.230 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:29:52.292 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:29:52.293 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.294 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:29:52.296 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.297 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:29:52.328 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:29:52.366 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:29:52.367 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:29:52.367 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:29:52.399 INFO 2458 --- [host-check-clockdiff3,64e1c9277144fc3a,52e2df4b53b62648] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535792 0 0 , error= 2025-10-27 11:29:52.407 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:29:52.407 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535202503000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:29:52.408 INFO 2458 --- [ocp-schedules-12,398cd241a3b9f57f,bc25b6166412b7eb] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:29:52.626 WARN 2458 --- [ocp-async-9,c174f26d1d805794,6d848cb76abf9d72] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:52.626 INFO 2458 --- [ocp-async-2,7d6b616bdf286a2a,3618d4ce02ec7292] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:53.164 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:29:53.235 INFO 2458 --- [ocp-schedules-9,b2096368769f5553,542e5afec478a314] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20390,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5074,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7888} 2025-10-27 11:29:53.235 INFO 2458 --- [ocp-schedules-9,b2096368769f5553,542e5afec478a314] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:29:53.707 INFO 2458 --- [prometheus-minute-schedule-1,83c51eeeb2404ba5,ca011e49a2d2cc7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:53.714 INFO 2458 --- [prometheus-minute-schedule-1,83c51eeeb2404ba5,ca011e49a2d2cc7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:54.510 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:54.511 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:54.626 INFO 2458 --- [ocp-async-4,a01fc93b1ec2d291,a60377bd805fda6a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:54.626 WARN 2458 --- [ocp-async-7,a957ee6c02d7d614,3f767fa8309165ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:55.323 INFO 2458 --- [ocp-schedules-27,866dcac4a1120e93,8825283daa3078f7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:29:55.326 INFO 2458 --- [ocp-schedules-27,866dcac4a1120e93,8825283daa3078f7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:29:55.467 INFO 2458 --- [ocp-schedules-28,6591aa3f8672c8b0,a99ddbcff858db35] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:29:56.626 INFO 2458 --- [ocp-async-4,d13886d29ab870b1,0bd55c207d89c9a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:56.626 WARN 2458 --- [ocp-async-1,30a199fea77dec7c,deec76e85419d0b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:56.707 INFO 2458 --- [prometheus-minute-schedule-1,b0930f648d96d86c,adfc7a9ffc9e84f0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:56.714 INFO 2458 --- [prometheus-minute-schedule-1,b0930f648d96d86c,adfc7a9ffc9e84f0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:29:56.906 INFO 2458 --- [ocp-schedules-20,5768750ed9cbc675,91a74d38c343e056] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:29:57.511 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:29:57.513 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:29:57.743 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7ec5a24ad341e660,d8d38335916056d6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=7ec5a24ad341e660, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:29:57.743 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7ec5a24ad341e660,d8d38335916056d6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=7ec5a24ad341e660, duration=2 ms] 2025-10-27 11:29:58.294 INFO 2458 --- [ocp-schedules-7,1fd365e8aff8eef7,6554762c0fbb3dbc] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:29:58.301 INFO 2458 --- [ocp-schedules-7,1fd365e8aff8eef7,6554762c0fbb3dbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:29:58.352 INFO 2458 --- [ocp-schedules-7,1fd365e8aff8eef7,6554762c0fbb3dbc] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:29:58.537 INFO 2458 --- [ocp-async-4,feef10770cb83994,74b25882de62d00f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:29:58.626 INFO 2458 --- [ocp-async-9,1b71532c113f1c89,ccc74c35e7669e8a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:29:58.626 WARN 2458 --- [ocp-async-2,9f0af73649e75202,0640e437ed350388] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:29:59.490 INFO 2458 --- [ocp-schedules-29,a866e8045b46dbf6,5b1afab254125a2d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:29:59.707 INFO 2458 --- [prometheus-minute-schedule-1,91e2474e16a27ea7,d628eeb085b0cba3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535800, interval=60 2025-10-27 11:29:59.714 INFO 2458 --- [prometheus-minute-schedule-1,91e2474e16a27ea7,d628eeb085b0cba3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:00.164 INFO 2458 --- [ocp-schedules-8,903bc409b2bec977,1d93894e940d9906] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:00.212 WARN 2458 --- [scanner-112198247-1,903bc409b2bec977,102f44abb112f137] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:00.510 INFO 2458 --- [alarm-detect-12,903bc409b2bec977,943dd1241b22d590] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535770 ~ 1761535790 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535785, value=60.29)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535785, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535785=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:30:00.513 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:00.513 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:00.626 INFO 2458 --- [ocp-async-9,0b7c1c50c7c7325e,f561482ec5be2a36] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:00.626 WARN 2458 --- [ocp-async-10,72d9cd3a558b1348,28c534e026e302e3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:00.887 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:25:01.0, nextRun=2025-10-27 11:30:00.0, currentDate=Mon Oct 27 11:30:00 CST 2025 2025-10-27 11:30:00.888 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:25:01.0, nextRunTime=2025-10-27 11:30:00.0} 2025-10-27 11:30:00.908 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:30:00.908 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:30:00.908+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:30:00.938 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:25:01.0, nextRun=2025-10-27 11:30:00.0, currentDate=Mon Oct 27 11:30:00 CST 2025 2025-10-27 11:30:00.939 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:25:01.0, nextRunTime=2025-10-27 11:30:00.0} 2025-10-27 11:30:00.999 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:30:01.033 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:30:01.040 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:30:01.040 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,9926fc0d385efc06] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:30:01.040+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:30:01.076 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,6ad115d7041da4ea] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9176, operation:EXECUTE, state: RUNNING 2025-10-27 11:30:01.114 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:30:01.115 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9176, context=Context{parallelIdx=-1, stringMap={task_instance_id=9108, task_operation=execute, latest_execution_start_time=2025-10-27T11:30:01.051+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9176}, listMap={}}, executor=10.40.65.146 2025-10-27 11:30:01.177 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.287 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.349 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.414 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.557 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.670 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:01.715354+08:00, lastProcessTime=2025-10-27T11:29:51.713053+08:00 2025-10-27 11:30:01.897 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.964 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:01.715354+08:00 2025-10-27 11:30:01.965 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:30:01.984 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:30:01.984 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:30:01.991 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.010 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.026 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.053 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.095 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.098 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,90ba04955d07d3d7] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9177, operation:EXECUTE, state: RUNNING 2025-10-27 11:30:02.127 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:30:02.131 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9177, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9109, task_operation=execute, latest_execution_start_time=2025-10-27T11:30:02.095+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9177}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:30:02.131 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:30:02.139 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.225 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.245 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:30:02.256 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:30:02.257 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:30:02.270 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,00289565031ece84] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9176, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:30:02.285 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:30:02.299 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:30:02.445 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9108, status:SUCCESS 2025-10-27 11:30:02.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:30:02.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:30:02.477 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9108, status:SUCCESS 2025-10-27 11:30:02.502 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,1e65abc12aa8b648,42612d349f64516f] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&version=2&ObClusterId=1760510187, client=10.40.65.146, traceId=1e65abc12aa8b648, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase,1760510187,2,{"ObRegion":"myoceanbase","ObCluster":"myoceanbase","ObRegionId":1760510187,"ObClusterId":1760510187,"Type":"PRIMARY","timestamp":1761535802501312,"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:30:02.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,1e65abc12aa8b648,42612d349f64516f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase, obClusterId=1760510187, rootServerJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, groupName=myoceanbase, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:30:02.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,1e65abc12aa8b648,42612d349f64516f] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:30:02.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,1e65abc12aa8b648,42612d349f64516f] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase, clusterId=1760510187, affectRows=1 2025-10-27 11:30:02.526 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,1e65abc12aa8b648,42612d349f64516f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.65.146, traceId=1e65abc12aa8b648, duration=25 ms] 2025-10-27 11:30:02.588 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:30:02.591 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:30:02.591 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:30:02.626 WARN 2458 --- [ocp-async-1,cd0f99d7b026c755,006961c0490061c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:02.626 INFO 2458 --- [ocp-async-6,b64cb9e72e262c48,d1145909b488f766] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:02.707 INFO 2458 --- [prometheus-minute-schedule-1,8918a33cc900d189,fb4e742b246b57b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:02.718 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:30:02.721 INFO 2458 --- [prometheus-minute-schedule-1,8918a33cc900d189,fb4e742b246b57b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:02.724 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.perf.sql.internal.SqlParseService : parse 1 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:30:02.853 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,a3e17a583a03bba5] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9177, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:30:03.501 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9109, status:SUCCESS 2025-10-27 11:30:03.503 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9109, status:SUCCESS 2025-10-27 11:30:03.514 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:03.514 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:04.233 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,1c35ffbd03e9a3f1] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:20:05.0, nextRun=2025-10-27 11:30:04.0, currentDate=Mon Oct 27 11:30:04 CST 2025 2025-10-27 11:30:04.233 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,1c35ffbd03e9a3f1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-10-27 11:20:05.0, nextRunTime=2025-10-27 11:30:04.0} 2025-10-27 11:30:04.261 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,1c35ffbd03e9a3f1] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-10-27 11:30:04.261 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,1c35ffbd03e9a3f1] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:30:04.261+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:30:04.401 INFO 2458 --- [ocp-schedules-11,310d27d34a7cbfe7,d3a8e4dc2edc3273] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:30:04.409 INFO 2458 --- [ocp-schedules-11,310d27d34a7cbfe7,d3a8e4dc2edc3273] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:04.448 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:04.449 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:30:04.462 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:04.462 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:04.462 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:04.462 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:04.462 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:04.463 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:30:04.477 INFO 2458 --- [ocp-schedules-11,310d27d34a7cbfe7,d3a8e4dc2edc3273] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:30:04.626 WARN 2458 --- [ocp-async-6,32ab4875b8c77030,addb37d5a79dd571] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:04.627 INFO 2458 --- [ocp-async-6,e5063d948d970150,d0d05fba3e475be2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:05.153 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,a1ceba2307a1d64a] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9178, operation:EXECUTE, state: RUNNING 2025-10-27 11:30:05.204 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:30:05.205 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9178, context=Context{parallelIdx=-1, stringMap={task_instance_id=9110, task_operation=execute, latest_execution_start_time=2025-10-27T11:30:05.148+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=9178}, listMap={}}, executor=10.40.65.146 2025-10-27 11:30:05.214 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:30:05.214 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:30:05.215 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.216 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:30:05.224 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.224 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.226 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.226 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:30:05.240 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.241 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.242 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.242 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:30:05.248 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:30:05.248 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:30:05.250 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.250 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:30:05.252 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.253 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.254 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.254 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:30:05.274 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.274 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.275 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.276 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:30:05.291 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.292 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.293 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.293 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:30:05.308 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.308 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.309 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.310 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:30:05.327 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.327 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.328 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.329 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:30:05.334 INFO 2458 --- [ocp-schedules-23,d7efa2c6a81a69e7,a5642f8b0b8ab319] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:05.337 INFO 2458 --- [ocp-schedules-23,d7efa2c6a81a69e7,a5642f8b0b8ab319] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:05.342 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.343 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.344 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.345 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:30:05.358 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.359 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.360 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.361 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:30:05.374 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.375 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.375 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.376 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:30:05.383 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.384 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.385 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.385 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:30:05.393 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.393 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.394 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.395 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:30:05.400 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.401 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.402 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.402 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:30:05.408 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.410 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:30:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.418 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.419 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:30:05.426 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.427 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.427 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.428 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:30:05.435 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.435 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.436 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.436 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:30:05.444 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.444 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.445 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.446 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:30:05.451 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.452 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.453 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.453 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:30:05.459 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.460 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.461 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.461 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:30:05.470 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.471 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.471 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.472 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:30:05.482 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.482 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.483 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.484 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:30:05.496 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.497 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.498 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.498 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:30:05.504 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.505 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.506 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.506 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:30:05.513 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.514 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.515 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.515 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:30:05.521 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.522 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.523 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.523 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:30:05.529 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.530 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.530 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.531 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:30:05.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.538 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.539 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:30:05.544 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.544 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.545 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.546 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:30:05.553 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.553 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.554 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.555 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:30:05.561 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.561 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.562 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.563 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:30:05.570 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.570 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.571 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.572 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:30:05.577 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.578 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:30:05.585 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.586 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.586 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.587 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:30:05.592 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.593 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.594 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.594 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:30:05.599 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.600 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.601 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.602 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:30:05.608 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.608 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.609 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.610 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:30:05.616 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.617 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.618 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.618 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:30:05.624 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.624 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.625 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.626 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:30:05.631 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.631 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.632 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.633 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:30:05.638 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.639 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.640 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.640 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:30:05.646 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.646 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.647 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.648 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:30:05.653 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.654 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.654 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.655 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:30:05.660 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.660 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.661 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.662 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:30:05.670 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.670 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.671 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.672 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:30:05.679 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.679 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.680 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.681 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:30:05.687 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.687 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.688 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.689 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:30:05.694 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.694 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.695 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.696 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:30:05.702 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.702 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.703 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.703 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:30:05.707 INFO 2458 --- [prometheus-minute-schedule-1,d7db200818280b0d,6a88daa93f427f8a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:05.708 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.709 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.710 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.710 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:30:05.715 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.716 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.717 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.717 INFO 2458 --- [prometheus-minute-schedule-1,d7db200818280b0d,6a88daa93f427f8a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:05.717 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:30:05.728 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.728 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.729 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.730 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:30:05.735 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.735 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.736 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.737 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:30:05.743 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.743 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.744 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.745 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:30:05.750 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.750 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.751 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.752 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:30:05.757 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.758 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.760 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.760 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:30:05.767 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.767 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.768 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.768 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:30:05.773 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.773 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.774 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.775 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:30:05.780 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.781 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:05.782 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:05.782 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:30:05.786 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:30:05.803 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,3d43850af94a6462] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9178, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:30:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:30:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:30:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:30:06.061 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:30:06.101 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:30:06.129 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:30:06.129 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:30:06.129 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:30:06.129 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:30:06.133 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:06.134 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:30:06.136 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:06.136 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:30:06.217 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:27:30, start=2025-10-27T11:25:00, end=2025-10-27T11:27:30, updates=250, granule=120000ms, delay=150000ms 2025-10-27 11:30:06.224 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:27:30, target=2025-10-27T11:27:30, granule=120000ms, delay=0ms) 2025-10-27 11:30:06.311 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - source=2025-10-27T11:27:30, start=2025-10-27T11:12:30, end=2025-10-27T11:22:30, updates=297, granule=600000ms, delay=900000ms 2025-10-27 11:30:06.328 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=1) - not enough cached interval (source=2025-10-27T11:27:30, target=2025-10-27T11:22:30, granule=600000ms, delay=300000ms) 2025-10-27 11:30:06.328 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=324ms, period=120000ms 2025-10-27 11:30:06.514 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:06.515 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:06.553 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9110, status:SUCCESS 2025-10-27 11:30:06.554 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9110, status:SUCCESS 2025-10-27 11:30:06.627 WARN 2458 --- [ocp-async-6,c70a8cb0c48d53e7,d853e5aef25e8d71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:06.627 INFO 2458 --- [ocp-async-7,470f9757b7c33f14,093b5d81f8e84bb1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:06.912 INFO 2458 --- [ocp-schedules-5,97e143bb015d4f9c,bfd128de2ca97409] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:07.576 INFO 2458 --- [ocp-schedules-2,9c00e6de616ce93f,c8875ec0c1c0f9df] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:30:07.584 INFO 2458 --- [ocp-schedules-2,9c00e6de616ce93f,c8875ec0c1c0f9df] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:07.584 INFO 2458 --- [ocp-schedules-2,9c00e6de616ce93f,c8875ec0c1c0f9df] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:08.537 INFO 2458 --- [ocp-async-1,b5b87efea138d1fd,f370e25aa7f6b153] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:08.627 WARN 2458 --- [ocp-async-4,78e3d38457524638,1655b7ba149f4ec4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:08.627 INFO 2458 --- [ocp-async-3,4d15eb387b7ec582,e5f6b1c121aef27f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:08.707 INFO 2458 --- [prometheus-minute-schedule-1,2a3fe4fa5e39d880,66311a346d8f456a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:08.715 INFO 2458 --- [prometheus-minute-schedule-1,2a3fe4fa5e39d880,66311a346d8f456a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:09.516 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:09.516 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:10.014 INFO 2458 --- [ocp-schedules-25,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:30:10.014 INFO 2458 --- [ocp-schedules-13,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:30:10.053 INFO 2458 --- [Check obproxy1,be89802bcb5f03a5,d92795737dcc389d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:30:10.054 INFO 2458 --- [Check obproxy1,be89802bcb5f03a5,d92795737dcc389d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:30:10.057 INFO 2458 --- [Collect obproxy1,3ac68c34b5a455da,a54923629e05bdbb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:30:10.057 INFO 2458 --- [Collect obproxy1,3ac68c34b5a455da,a54923629e05bdbb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:30:10.084 INFO 2458 --- [Collect obproxy1,3ac68c34b5a455da,a54923629e05bdbb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:30:10.087 INFO 2458 --- [Collect obproxy1,3ac68c34b5a455da,a54923629e05bdbb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:30:10.087 INFO 2458 --- [Check obproxy1,be89802bcb5f03a5,d92795737dcc389d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:30:10.090 INFO 2458 --- [Check obproxy1,be89802bcb5f03a5,d92795737dcc389d] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:30:10.090 INFO 2458 --- [Check obproxy1,be89802bcb5f03a5,d92795737dcc389d] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:30:10.090 INFO 2458 --- [Collect obproxy1,3ac68c34b5a455da,a54923629e05bdbb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:30:10.186 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:30:10.521 INFO 2458 --- [ocp-schedules-15,47cf78e761610fe6,922216debc4e0f9f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:10.536 WARN 2458 --- [scanner-112198690-1,47cf78e761610fe6,9d9063b4f83ed796] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:10.627 WARN 2458 --- [ocp-async-9,69cf2d24b9bc42f7,48ac67663760fa1f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:10.627 INFO 2458 --- [ocp-async-9,bc3d3b9d7bd675db,a03ff228589c352e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:10.653 INFO 2458 --- [alarm-detect-9,47cf78e761610fe6,272592d4dd6af6e3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535785 ~ 1761535800 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535800, value=58.664)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535800, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535800=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:30:11.707 INFO 2458 --- [prometheus-minute-schedule-1,f892ff024bad13b9,8196e7372281bd47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:11.717 INFO 2458 --- [prometheus-minute-schedule-1,f892ff024bad13b9,8196e7372281bd47] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:11.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:11.716466+08:00, lastProcessTime=2025-10-27T11:30:01.715354+08:00 2025-10-27 11:30:12.128 INFO 2458 --- [metric-archive-async-3,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535740, nodeCount=817, affectRows=817 2025-10-27 11:30:12.203 INFO 2458 --- [ocp-schedules-11,0f4edc5afe66ec94,6735c327bd79ed5c] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:30:12.209 INFO 2458 --- [ocp-schedules-11,0f4edc5afe66ec94,6735c327bd79ed5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:12.254 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:11.716466+08:00 2025-10-27 11:30:12.304 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:30:12.305 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.308 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:30:12.319 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.320 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.338 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:30:12.338 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.343 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:30:12.348 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.348 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.359 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.359 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.376 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.377 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.379 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.379 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.386 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.387 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.405 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.406 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.416 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.417 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:30:12.419 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.420 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.432 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.433 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.452 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.453 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.467 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.468 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.468 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.470 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.470 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.471 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.472 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.473 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.474 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.475 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.475 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.476 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.477 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:30:12.483 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.483 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:30:12.485 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:30:12.486 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.487 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:30:12.516 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:12.517 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:12.521 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.521 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:30:12.536 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:12.536 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.537 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:30:12.547 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.547 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:30:12.547 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.547 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:30:12.549 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.550 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:30:12.550 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:30:12.550 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:30:12.552 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.552 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:30:12.581 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.581 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:30:12.609 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.610 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:30:12.627 WARN 2458 --- [ocp-async-5,6ccdee277257ee3d,a1f0c804a08803c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:12.627 INFO 2458 --- [ocp-async-9,8f4accfe9f2c5cac,f1267842310ce6e1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:12.658 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.659 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.661 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.662 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.662 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.663 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.664 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.665 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.666 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.667 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.667 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.668 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.668 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.669 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.670 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.671 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.671 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.672 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.673 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.674 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:30:12.675 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:30:12.676 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.677 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:30:12.677 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:30:12.680 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:30:12.782 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:12.783 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:12.783 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:12.783 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:12.783 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:12.783 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:12.784 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:30:12.811 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:30:12.815 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:30:12.815 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:30:12.815 INFO 2458 --- [ocp-schedules-11,0f4edc5afe66ec94,6735c327bd79ed5c] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:30:12.996 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587947) Duplicate entry '835370603008958464-1761535680' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587947) Duplicate entry '835370603008958464-1761535680' for key 'PRIMARY' 2025-10-27 11:30:13.070 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=74, affectRows=74 2025-10-27 11:30:13.070 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=73, validCount=73 2025-10-27 11:30:13.086 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=1000, affectRows=1000 2025-10-27 11:30:13.090 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=994, validCount=992 2025-10-27 11:30:13.106 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=1000, affectRows=1000 2025-10-27 11:30:13.110 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=996, validCount=992 2025-10-27 11:30:13.112 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=1000, affectRows=1000 2025-10-27 11:30:13.116 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=999, validCount=995 2025-10-27 11:30:13.121 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=1000, affectRows=1000 2025-10-27 11:30:13.125 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=996, validCount=992 2025-10-27 11:30:14.623 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,946748e2ebd02b5b,9d9f2b68fb6071ed] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=946748e2ebd02b5b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:30:14.623 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,946748e2ebd02b5b,9d9f2b68fb6071ed] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=946748e2ebd02b5b, duration=1 ms] 2025-10-27 11:30:14.627 WARN 2458 --- [ocp-async-8,d60d26091db47678,313795e08c0324e2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:14.627 INFO 2458 --- [ocp-async-4,4d98db630a9d11ef,19a34b87c8f5b002] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:14.707 INFO 2458 --- [prometheus-minute-schedule-1,a8b55934747d4439,f870928685f4d75d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:14.717 INFO 2458 --- [prometheus-minute-schedule-1,a8b55934747d4439,f870928685f4d75d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:15.345 INFO 2458 --- [ocp-schedules-13,095e51a9e3d91207,cb24b18e08932eca] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:15.350 INFO 2458 --- [ocp-schedules-13,095e51a9e3d91207,cb24b18e08932eca] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:15.517 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:15.519 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:15.946 INFO 2458 --- [ocp-schedules-21,3aea8c9725f5f0dc,1920c651fd2563cc] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:30:15.946+08:00, clusters=[1, 2] 2025-10-27 11:30:15.951 INFO 2458 --- [ocp-schedules-21,3aea8c9725f5f0dc,1920c651fd2563cc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:30:15.954 INFO 2458 --- [ocp-schedules-21,3aea8c9725f5f0dc,1920c651fd2563cc] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:30:16.627 INFO 2458 --- [ocp-async-2,05a8beb14b1aeee4,1d7ef72f559774aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:16.627 WARN 2458 --- [ocp-async-10,397bae2ac28e8669,770fd1d82af1a326] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:16.927 INFO 2458 --- [ocp-schedules-22,81b39f953d8d2db2,6e3a035ec86b3b9a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:17.700 INFO 2458 --- [ocp-schedules-26,1417816e5528a12f,3d2c3cfc8d3fdb0a] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:30:17.700 INFO 2458 --- [ocp-schedules-26,1417816e5528a12f,3d2c3cfc8d3fdb0a] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:30:17.707 INFO 2458 --- [prometheus-minute-schedule-1,f06d767a912f7888,3ac8f966879cd189] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:17.714 INFO 2458 --- [prometheus-minute-schedule-1,f06d767a912f7888,3ac8f966879cd189] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:17.771 INFO 2458 --- [Collect all tenants major compactions2,1417816e5528a12f,f3e64427dbc17ac3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:30:17.771 INFO 2458 --- [Collect all tenants major compactions1,1417816e5528a12f,5c104bbb7a5c544f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:30:17.771 INFO 2458 --- [Collect all tenants major compactions3,1417816e5528a12f,91099256d456551d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:30:17.771 INFO 2458 --- [Collect all tenants major compactions4,1417816e5528a12f,bee7c0553eff8ef0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:30:17.774 INFO 2458 --- [Collect all tenants major compactions6,1417816e5528a12f,e5ef1b3930e8e744] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:30:17.775 INFO 2458 --- [Collect all tenants major compactions5,1417816e5528a12f,6f5c34db82dc02e6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:30:17.812 INFO 2458 --- [Collect all tenants major compactions6,1417816e5528a12f,e5ef1b3930e8e744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.814 INFO 2458 --- [Collect all tenants major compactions6,1417816e5528a12f,e5ef1b3930e8e744] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:30:17.815 INFO 2458 --- [Collect all tenants major compactions5,1417816e5528a12f,6f5c34db82dc02e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.817 INFO 2458 --- [Collect all tenants major compactions5,1417816e5528a12f,6f5c34db82dc02e6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:30:17.825 INFO 2458 --- [Collect all tenants major compactions6,1417816e5528a12f,e5ef1b3930e8e744] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:30:17.826 INFO 2458 --- [Collect all tenants major compactions5,1417816e5528a12f,6f5c34db82dc02e6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:30:17.832 INFO 2458 --- [Collect all tenants major compactions4,1417816e5528a12f,bee7c0553eff8ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.834 INFO 2458 --- [Collect all tenants major compactions4,1417816e5528a12f,bee7c0553eff8ef0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:30:17.834 INFO 2458 --- [Collect all tenants major compactions2,1417816e5528a12f,f3e64427dbc17ac3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.835 INFO 2458 --- [Collect all tenants major compactions3,1417816e5528a12f,91099256d456551d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.835 INFO 2458 --- [Collect all tenants major compactions2,1417816e5528a12f,f3e64427dbc17ac3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:30:17.836 INFO 2458 --- [Collect all tenants major compactions3,1417816e5528a12f,91099256d456551d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:30:17.837 INFO 2458 --- [Collect all tenants major compactions4,1417816e5528a12f,bee7c0553eff8ef0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:30:17.839 INFO 2458 --- [Collect all tenants major compactions1,1417816e5528a12f,5c104bbb7a5c544f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:17.839 INFO 2458 --- [Collect all tenants major compactions2,1417816e5528a12f,f3e64427dbc17ac3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:30:17.840 INFO 2458 --- [Collect all tenants major compactions3,1417816e5528a12f,91099256d456551d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:30:17.840 INFO 2458 --- [Collect all tenants major compactions1,1417816e5528a12f,5c104bbb7a5c544f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:30:17.844 INFO 2458 --- [Collect all tenants major compactions1,1417816e5528a12f,5c104bbb7a5c544f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:30:17.846 INFO 2458 --- [ocp-schedules-26,1417816e5528a12f,3d2c3cfc8d3fdb0a] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:30:17.855 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535740, nodeCount=1000, affectRows=1000 2025-10-27 11:30:17.859 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535740, nodeCount=1000, validCount=988 2025-10-27 11:30:18.519 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:18.520 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:18.537 INFO 2458 --- [ocp-async-2,9550b98084773d93,3465ea6e2798b5c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:18.628 INFO 2458 --- [ocp-async-7,dc3e6cf679cc5170,1dee431b670e9fc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:18.628 WARN 2458 --- [ocp-async-9,1ad5ffbd85f2199c,910b9a88217f3646] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:19.005 INFO 2458 --- [ocp-schedules-18,67b1de5b3d0b6295,eb4b8d6fe770e5e3] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:19.005 INFO 2458 --- [ocp-schedules-18,67b1de5b3d0b6295,eb4b8d6fe770e5e3] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:30:20.470 INFO 2458 --- [ocp-schedules-10,19a38690b0d19f6f,c97c849bc1ef21e5] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:30:20.628 WARN 2458 --- [ocp-async-1,385f222e7c25d1ce,9240d8fe1a8518c7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:20.628 INFO 2458 --- [ocp-async-3,abf89ab83aa9559c,1fd244e8fbd3b198] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:20.659 INFO 2458 --- [ocp-schedules-9,d414b9c3f9cea7dd,d2efa01c5cbd47fa] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:20.707 INFO 2458 --- [prometheus-minute-schedule-1,6af116b66aa4fbf2,983153818cdf4b70] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:20.713 INFO 2458 --- [prometheus-minute-schedule-1,6af116b66aa4fbf2,983153818cdf4b70] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:20.751 WARN 2458 --- [scanner-112200429-1,d414b9c3f9cea7dd,9c23b658611b3ce5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:21.411 INFO 2458 --- [ocp-schedules-3,1e15bb1e483435e2,ea84b3a86dc2af7a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:21.411 INFO 2458 --- [ocp-schedules-21,d7a3b9db9f76fe07,fd5cdbb30bcb0bea] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:30:21.412 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:30:21.413 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:30:21.413 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:30:21.413 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:30:21.413 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:30:21.428 INFO 2458 --- [ocp-schedules-1,42e33de1f9e8bc69,e6360f0f724a54df] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:21.428 INFO 2458 --- [host-check-clockdiff1,42e33de1f9e8bc69,b35cd6ea08b2b0bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:30:21.428 INFO 2458 --- [host-check-clockdiff2,42e33de1f9e8bc69,46acec8d81292202] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:30:21.428 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:21.428 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:21.429 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:21.429 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-clockdiff3,42e33de1f9e8bc69,f88bedcb3544b2ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-clockdiff5,42e33de1f9e8bc69,c8f1ac3f59261edd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-clockdiff4,42e33de1f9e8bc69,7904e40ba2ef5c40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:30:21.430 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:21.434 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:21.435 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:21.435 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:21.435 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:21.435 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:21.436 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:21.436 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:21.437 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:21.437 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:21.437 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:21.445 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:21.448 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:21.446 INFO 2458 --- [ocp-schedules-13,b498d2974236fdf9,f4d3a3a3eb521660] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:30:21.448 INFO 2458 --- [host-check-clockdiff4,42e33de1f9e8bc69,7904e40ba2ef5c40] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535821 0 0 , error= 2025-10-27 11:30:21.448 INFO 2458 --- [host-check-clockdiff2,42e33de1f9e8bc69,46acec8d81292202] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535821 -1 -1 , error= 2025-10-27 11:30:21.457 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:21.458 INFO 2458 --- [host-check-clockdiff5,42e33de1f9e8bc69,c8f1ac3f59261edd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535821 1 1 , error= 2025-10-27 11:30:21.458 INFO 2458 --- [host-check-clockdiff3,42e33de1f9e8bc69,f88bedcb3544b2ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535821 0 0 , error= 2025-10-27 11:30:21.461 INFO 2458 --- [host-check-clockdiff1,42e33de1f9e8bc69,b35cd6ea08b2b0bd] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535821 0 0 , error= 2025-10-27 11:30:21.512 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:30:21.514 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:30:21.514182653+08:00 2025-10-27 11:30:21.515 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:30:21.515 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:30:21.514+08:00 2025-10-27 11:30:21.515 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:30:21.517 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:30:21.518 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:30:21.518 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:30:21.518 INFO 2458 --- [host-check-agent-status1,1e15bb1e483435e2,813f28b5e58d2764] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:30:21.519 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:30:21.519 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:30:21.519 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:30:21.519 INFO 2458 --- [host-check-agent-status2,1e15bb1e483435e2,d05938b44b561d18] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:30:21.520 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:30:21.518779521+08:00 2025-10-27 11:30:21.520 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:21.520 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:30:21.520+08:00 2025-10-27 11:30:21.520 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:30:21.521 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:30:21.521 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:21.521 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:30:21.521 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:30:21.521 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:30:21.521 INFO 2458 --- [host-check-agent-status3,1e15bb1e483435e2,fb3d2f11749e0bc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:30:21.522 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:30:21.521798962+08:00 2025-10-27 11:30:21.523 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:30:21.514+08:00 and status AVAILABLE 2025-10-27 11:30:21.523 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:30:21.522+08:00 2025-10-27 11:30:21.523 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:30:21.522846877+08:00 2025-10-27 11:30:21.524 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:30:21.524 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:30:21.524 INFO 2458 --- [host-check-agent-status5,1e15bb1e483435e2,5486741808466d56] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:30:21.524 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:30:21.523+08:00 2025-10-27 11:30:21.525 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:30:21.525 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:30:21.526 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:30:21.526213688+08:00 2025-10-27 11:30:21.526 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:30:21.526+08:00 2025-10-27 11:30:21.527 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:30:21.527 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:30:21.527 INFO 2458 --- [host-check-agent-status4,1e15bb1e483435e2,382996bc5b490dbb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:30:21.527 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:30:21.520+08:00 and status AVAILABLE 2025-10-27 11:30:21.528 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:30:21.530 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:30:21.522+08:00 and status AVAILABLE 2025-10-27 11:30:21.531 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:30:21.533 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:30:21.523+08:00 and status AVAILABLE 2025-10-27 11:30:21.533 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:30:21.526+08:00 and status AVAILABLE 2025-10-27 11:30:21.534 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:30:21.537 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:30:21.539 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:30:21.545 INFO 2458 --- [ocp-schedules-29,582970d65dd2913f,fa8b02fd5b0ae13e] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:30:21.549 INFO 2458 --- [ocp-schedules-24,b7ee236b3b523617,110a435adb28b639] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:30:21.549 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:21.550 INFO 2458 --- [ocp-schedules-29,582970d65dd2913f,fa8b02fd5b0ae13e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:21.552 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:21.552 INFO 2458 --- [ocp-schedules-24,b7ee236b3b523617,110a435adb28b639] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:21.553 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:21.553 INFO 2458 --- [ocp-schedules-24,b7ee236b3b523617,110a435adb28b639] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:30:21.554 INFO 2458 --- [ocp-schedules-10,ea5e83f2e10afd8e,f7856711d3a149f1] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:30:21.555 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:21.557 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:21.558 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:21.558 INFO 2458 --- [ocp-schedules-10,ea5e83f2e10afd8e,f7856711d3a149f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:21.560 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:21.561 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:21.564 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:21.565 INFO 2458 --- [ocp-schedules-20,50d290eb7a7dde7a,87ab5e10d63dfc27] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:30:21.567 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:21.595 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.596 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:21.597 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:30:21.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.632 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:21.633 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:21.634 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:21.637 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63} 2025-10-27 11:30:21.638 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:21.640 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:30:21.641 INFO 2458 --- [ocp-schedules-29,582970d65dd2913f,fa8b02fd5b0ae13e] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:30:21.648 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.649 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:21.651 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterVi 2025-10-27 11:30:21.651 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:21.653 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:30:21.659 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 3 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:21.661 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:30:21.666 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:30:21.670 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:21.676 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:21.686 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:30:21.693 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:30:21.693 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:30:21.699 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:30:21.700 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:30:21.704 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:30:21.704 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:21.714056+08:00, lastProcessTime=2025-10-27T11:30:11.716466+08:00 2025-10-27 11:30:21.720 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.720 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:21.745 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:30:21.747 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:21.748 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:21.748 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:21.748 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:21.748 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:30:21.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.749 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:30:21.768 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:30:21.769 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:30:21.784 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:30:21.797 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:21.799 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:21.813 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:30:21.817 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:30:21.819 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:30:21.819 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:30:21.828 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:30:21.828 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:21.829 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.829 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:30:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.848 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:21.849 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.851 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:30:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:30:21.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:21.868 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:21.869 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:30:21.871 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:21.871 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:21.879 INFO 2458 --- [ocp-schedules-30,1d2914cc2ef83743,5535620a2da1ab69] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:30:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:21.882 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449421882000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:21.882Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:21.888 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:30:21.891 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:30:21.898 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:30:21.898 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:21.900 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:21.900 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449421900000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:21.900Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:21.901 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:21.904 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:30:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:21.914 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:30:21.918 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:30:21.926 INFO 2458 --- [host-check-host-status3,d7a3b9db9f76fe07,2b535260f1555c64] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:30:21.929 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:21.930 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:21.930 INFO 2458 --- [host-check-host-status2,d7a3b9db9f76fe07,2e5b86b991f35959] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:30:21.940 INFO 2458 --- [host-check-host-status1,d7a3b9db9f76fe07,8c990b07952aa2e3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:30:21.941 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2025-10-27 11:30:21.941 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:21.941 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:21.941 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:21.944 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.944 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:21.946 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:21.946 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.946 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:21.947 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:21.948 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:30:21.949 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:30:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:21.953 INFO 2458 --- [host-check-host-status4,d7a3b9db9f76fe07,42ccf01754d1cd4a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:30:21.960 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:30:21.963 INFO 2458 --- [host-check-host-status5,d7a3b9db9f76fe07,674a7574ae90a48c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:30:21.973 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:21.974 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:21.974 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:21.974 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:21.974 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:21.976 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:21.979 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:30:21.979 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:21.882, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:21.982 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:21.982 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:21.997 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:21.998 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:22.007 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.007 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.012 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:30:22.032 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:30:22.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:22.042 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:22.045 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:21.714056+08:00 2025-10-27 11:30:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:22.076 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:22.076 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:30:22.077 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.077 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:30:22.079 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.080 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:30:22.102 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:30:22.118 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:22.118 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:22.126 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:22.126 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.127 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:30:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:22.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:22.154 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:30:22.154 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:30:22.194 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:22.239 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:22.240 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.240 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:30:22.255 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:30:22.260 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:21.9, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:22.260 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:22.262 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:22.262 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:30:22.262 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:22.264 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:22.265 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:22.266 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:22.267 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:30:22.267 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:22.267 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:22.269 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:22.270 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:22.270 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:30:22.270 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:22.273 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:22.277 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 4 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:22.290 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:30:22.290 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:30:22.401 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449422401000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=E9E2014C8CE705871C555597A6A32456, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:22.401Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:22.404 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449422404000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=CF05F891D761DB9505D1EA76FDB29165, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:22.404Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:22.408 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:30:22.408 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:22.409 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449422409000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=C9438F61ADA40409BBF0B756DEEDAE1C, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:22.409Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:22.412 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449422412000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=72BCB1753D053BFFF335341ABA02065F, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:30:22.412Z, endTime=2025-10-27T03:28:21.630Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:30:22.418 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 1 2025-10-27 11:30:22.419 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:22.434 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.434 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.434 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 104 2025-10-27 11:30:22.434 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:22.436 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 104 2025-10-27 11:30:22.436 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:30:22.440 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:22.440 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:22.450 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.450 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=CF05F891D761DB9505D1EA76FDB29165, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.458 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:22.458 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:22.461 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.461 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=72BCB1753D053BFFF335341ABA02065F, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.463 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.463 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=C9438F61ADA40409BBF0B756DEEDAE1C, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.465 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:30:22.467 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.467 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:30:22.469 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:22.469 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:22.470 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.470 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:30:22.470 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:30:22.470 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.470 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=E9E2014C8CE705871C555597A6A32456, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.471 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:30:22.475 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:30:22.475 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:22.401, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=6, sqlId=E9E2014C8CE705871C555597A6A32456, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:22.485 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.485 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=CF05F891D761DB9505D1EA76FDB29165, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.490 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:30:22.497 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.497 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=72BCB1753D053BFFF335341ABA02065F, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.499 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:30:22.499 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535701630000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=C9438F61ADA40409BBF0B756DEEDAE1C, sqlIdList=null, dynamicSql=false) 2025-10-27 11:30:22.502 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:30:22.504 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:30:22.505 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:22.404, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=6, sqlId=CF05F891D761DB9505D1EA76FDB29165, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:22.505 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:30:22.517 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:22.409, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=6, sqlId=C9438F61ADA40409BBF0B756DEEDAE1C, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:22.520 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:30:22.412, end=2025-10-27T11:28:21.63}, clusterId=2, tenantId=6, sqlId=72BCB1753D053BFFF335341ABA02065F, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:30:22.520 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:30:22.522 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:20.964, end=2025-10-27T11:28:21.63},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:30:22.522 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:30:22.522 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:30:22.524 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:30:22.546 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:22.547 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:30:22.581 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:30:22.581 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:30:22.581 INFO 2458 --- [ocp-schedules-10,ea5e83f2e10afd8e,f7856711d3a149f1] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:30:22.628 INFO 2458 --- [ocp-async-10,0cfa438efe7189ae,be2443dc3bb09fb4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:22.628 WARN 2458 --- [ocp-async-4,1bc61d405acc570c,199e028e866f8c6a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:23.707 INFO 2458 --- [prometheus-minute-schedule-1,18101c230c563e35,a670217e8e11f49e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:23.716 INFO 2458 --- [prometheus-minute-schedule-1,18101c230c563e35,a670217e8e11f49e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:24.521 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:24.522 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:24.628 WARN 2458 --- [ocp-async-6,7f127f2f0e67f175,8f059cec6cef80df] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:24.628 INFO 2458 --- [ocp-async-5,1dc06d41668e0bc9,8c89b5e2479d716a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:25.357 INFO 2458 --- [ocp-schedules-29,6bbaa1be569fd190,cb40af60c1775d3a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:25.361 INFO 2458 --- [ocp-schedules-29,6bbaa1be569fd190,cb40af60c1775d3a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:26.021 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,fea5b55c8cbdfa1e,7d4f8c6f75b9ea28] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/creators?null, client=10.40.198.13, traceId=fea5b55c8cbdfa1e, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.listCreatorInfo(TaskType), args=MANUAL,] 2025-10-27 11:30:26.022 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,a0c9a74d86042fbd,d9f2758baa003173] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/manual-instances?page=1&size=10, client=10.40.198.13, traceId=a0c9a74d86042fbd, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.TaskController.listManualTaskInstances(String,Pageable,List,String), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:30:26.028 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,fea5b55c8cbdfa1e,7d4f8c6f75b9ea28] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/creators client=10.40.198.13, traceId=fea5b55c8cbdfa1e, duration=14 ms] 2025-10-27 11:30:26.070 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,a0c9a74d86042fbd,d9f2758baa003173] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/manual-instances client=10.40.198.13, traceId=a0c9a74d86042fbd, duration=56 ms] 2025-10-27 11:30:26.628 WARN 2458 --- [ocp-async-7,7d03ac7d8822da68,b1876c6aff6124ca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:26.628 INFO 2458 --- [ocp-async-7,5be2cf32243f2449,ae5e8ad6ece5abf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:26.707 INFO 2458 --- [prometheus-minute-schedule-1,854cbc57ff246eb7,89047919efda2f47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:26.715 INFO 2458 --- [prometheus-minute-schedule-1,854cbc57ff246eb7,89047919efda2f47] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:26.846 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,177daf6fc4a60866,72a708e7c33d065d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=177daf6fc4a60866, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:30:26.866 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,177daf6fc4a60866,72a708e7c33d065d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=177daf6fc4a60866, duration=29 ms] 2025-10-27 11:30:26.936 INFO 2458 --- [ocp-schedules-20,930546c22a8ba777,c89fcd0bdc26f1ac] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:27.522 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:27.523 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:27.590 INFO 2458 --- [ocp-schedules-5,35c56a609ee91da9,9ebd2dcfc0314786] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:30:27.602 INFO 2458 --- [ocp-schedules-5,35c56a609ee91da9,9ebd2dcfc0314786] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:27.602 INFO 2458 --- [ocp-schedules-5,35c56a609ee91da9,9ebd2dcfc0314786] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:28.537 INFO 2458 --- [ocp-async-7,13071bd58e3b497f,ad46231e9e836c52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:28.628 WARN 2458 --- [ocp-async-4,85b2b1361e8a3303,e19eb140ccd59128] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:28.628 INFO 2458 --- [ocp-async-2,d8ec5ef525d15a0b,f57200aaa70cfce1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:29.025 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:30:29.025 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:30:29.030 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:30:29.030 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:30:29.035 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:30:29.035 INFO 2458 --- [ocp-schedules-16,61343e268ef3af24,b1e7824e83bd9785] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:30:29.483 INFO 2458 --- [ocp-schedules-24,ad4c8adffe70d353,6820d2f41b282068] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:30:29.492 INFO 2458 --- [ocp-schedules-24,ad4c8adffe70d353,6820d2f41b282068] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:30:29.492 INFO 2458 --- [ocp-schedules-24,ad4c8adffe70d353,6820d2f41b282068] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:30:29.707 INFO 2458 --- [prometheus-minute-schedule-1,91948a5a1eecbdd3,f1eeb0e84ce035ac] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:29.714 INFO 2458 --- [prometheus-minute-schedule-1,91948a5a1eecbdd3,f1eeb0e84ce035ac] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:30.523 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:30.524 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:30.628 WARN 2458 --- [ocp-async-6,060273b5b3f170cf,4e2db6b215bb321d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:30.628 INFO 2458 --- [ocp-async-9,c91503d154e2dfc4,ad24486ea45e30c4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:30.886 INFO 2458 --- [ocp-schedules-22,afb3a6e39017988b,8b3a0a2c5fd6bb3f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:30.921 WARN 2458 --- [scanner-112200659-1,afb3a6e39017988b,d901a7e818498ee9] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:31.264 INFO 2458 --- [alarm-detect-14,afb3a6e39017988b,58a3e47b5557d382] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535800 ~ 1761535821 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535815, value=71.47)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535815, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535815=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:30:31.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:31.713467+08:00, lastProcessTime=2025-10-27T11:30:21.714056+08:00 2025-10-27 11:30:31.956 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:31.713467+08:00 2025-10-27 11:30:32.274 INFO 2458 --- [ocp-schedules-10,7b142eb783ad6447,c627a0d7f3218504] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-10-27 11:30:32.274 INFO 2458 --- [ocp-schedules-10,7b142eb783ad6447,c627a0d7f3218504] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-10-27 11:30:32.274 INFO 2458 --- [ocp-schedules-10,7b142eb783ad6447,c627a0d7f3218504] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=8774 2025-10-27 11:30:32.526 INFO 2458 --- [ocp-schedules-10,7b142eb783ad6447,c627a0d7f3218504] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=8774 2025-10-27 11:30:32.629 WARN 2458 --- [ocp-async-8,193557dcf75c0768,ed3fb398d018f4a3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:32.629 INFO 2458 --- [ocp-async-8,73b7bce7d4aa1ebe,f3269b105a108b3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:32.707 INFO 2458 --- [prometheus-minute-schedule-1,32af0d9aecc0f0ca,b304decac81c1398] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:32.715 INFO 2458 --- [prometheus-minute-schedule-1,32af0d9aecc0f0ca,b304decac81c1398] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:33.524 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:33.525 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:33.950 INFO 2458 --- [ocp-schedules-10,a7fcc499e0b578ac,06482140e002e354] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:30:33.950 INFO 2458 --- [ocp-schedules-10,a7fcc499e0b578ac,06482140e002e354] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:30:34.429 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:30:34.429 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:30:34.450 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:30:34.450 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:30:34.450 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:30:34.495 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:30:34.495 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:30:34.495 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:30:34.514 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:30:34.514 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:30:34.514 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:30:34.523 INFO 2458 --- [ocp-schedules-7,77a2db97d9b32735,2d4a8c6716c3e2b4] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:30:34.629 INFO 2458 --- [ocp-async-10,b07de77c1b1ea082,7e11fbd1f874612b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:34.629 WARN 2458 --- [ocp-async-3,167ee6bb1bfea37c,9c3349c69dd27036] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:35.371 INFO 2458 --- [ocp-schedules-20,a7d06af3648b125d,bbfec903564a61a2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:35.374 INFO 2458 --- [ocp-schedules-20,a7d06af3648b125d,bbfec903564a61a2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:35.707 INFO 2458 --- [prometheus-minute-schedule-1,8a3e7ad799270836,7e938bca4f1fd0d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:35.715 INFO 2458 --- [prometheus-minute-schedule-1,8a3e7ad799270836,7e938bca4f1fd0d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:36.525 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:36.526 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:36.629 WARN 2458 --- [ocp-async-2,6070be638f4b00b2,778182071ec51bc9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:36.629 INFO 2458 --- [ocp-async-6,9c4da2136661cbe5,8066c3867d0ec808] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:36.942 INFO 2458 --- [ocp-schedules-10,a8d8593833c39eb7,c80c137e0fa07cc6] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:37.708 INFO 2458 --- [prometheus-cache-monitor1,1b169e530cb202d0,47e7429c2d06890e] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20367,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5078,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7888} 2025-10-27 11:30:37.708 INFO 2458 --- [prometheus-cache-monitor1,1b169e530cb202d0,47e7429c2d06890e] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:30:38.538 INFO 2458 --- [ocp-async-6,65818c484bf5d4d8,653187afc36fbfc6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:38.629 INFO 2458 --- [ocp-async-9,4e2ff20cf70f5499,1c125849b75ccb4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:38.629 WARN 2458 --- [ocp-async-7,6408bfe6794831e4,59fe7cc273435278] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:38.707 INFO 2458 --- [prometheus-minute-schedule-1,ee2138fe788c2310,6d03f1d78d87ab99] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:38.715 INFO 2458 --- [prometheus-minute-schedule-1,ee2138fe788c2310,6d03f1d78d87ab99] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:38.810 INFO 2458 --- [ocp-schedules-9,9bdc232b9113b998,6074c8744dd3f7a6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:30:38.816 INFO 2458 --- [ocp-schedules-9,9bdc232b9113b998,6074c8744dd3f7a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:38.816 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:30:38.816 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:30:38.817 INFO 2458 --- [ocp-schedules-9,9bdc232b9113b998,6074c8744dd3f7a6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:30:39.526 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:39.527 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:39.592 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:30:39.592 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:30:39.660 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:39.660 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:30:39.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:30:39.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:30:39.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:30:39.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:30:39.671 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:39.673 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:39.676 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:39.676 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:39.676 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:39.676 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:39.677 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:39.679 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:39.679 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:39.680 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:39.712 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:39.713 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:30:39.729 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:39.730 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:30:39.736 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:30:39.738 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:30:39.742 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:30:39.743 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:30:39.745 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:30:39.746 INFO 2458 --- [ocp-schedules-12,94cfcefcb9153189,b56c4ee8316632a6] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:30:39.790 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:39.791 INFO 2458 --- [ocp-schedules-27,3068eec654a62dbb,c55ea6f32425fa97] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:30:40.009 INFO 2458 --- [ocp-schedules-19,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:30:40.009 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:30:40.044 INFO 2458 --- [Check obproxy1,e329f988594f6ca2,5d57dc7625b50724] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:30:40.044 INFO 2458 --- [Check obproxy1,e329f988594f6ca2,5d57dc7625b50724] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:30:40.044 INFO 2458 --- [Collect obproxy1,662107844aafb077,bbaad5cbbcaf3c71] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:30:40.044 INFO 2458 --- [Collect obproxy1,662107844aafb077,bbaad5cbbcaf3c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:30:40.074 INFO 2458 --- [Collect obproxy1,662107844aafb077,bbaad5cbbcaf3c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:30:40.075 INFO 2458 --- [Check obproxy1,e329f988594f6ca2,5d57dc7625b50724] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:30:40.078 INFO 2458 --- [Collect obproxy1,662107844aafb077,bbaad5cbbcaf3c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:30:40.079 INFO 2458 --- [Check obproxy1,e329f988594f6ca2,5d57dc7625b50724] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:30:40.079 INFO 2458 --- [Check obproxy1,e329f988594f6ca2,5d57dc7625b50724] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:30:40.080 INFO 2458 --- [Collect obproxy1,662107844aafb077,bbaad5cbbcaf3c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:30:40.629 WARN 2458 --- [ocp-async-9,dd303a81855dc137,6ccc7a7d6d00c1ee] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:40.629 INFO 2458 --- [ocp-async-3,29b45ba7806cf581,a733e50a3edce42f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:41.274 INFO 2458 --- [ocp-schedules-5,aa054140238641b8,2e34bc76de984200] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:30:41.277 INFO 2458 --- [ocp-schedules-17,65a384722000ce7a,64a1915d52fb9247] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:41.300 WARN 2458 --- [scanner-112200843-1,65a384722000ce7a,84d5d58832bf6559] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:41.437 INFO 2458 --- [alarm-detect-15,65a384722000ce7a,3b33a53f95d352fc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535815 ~ 1761535831 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535830, value=65.188)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535830, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535830=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:30:41.707 INFO 2458 --- [prometheus-minute-schedule-1,15f2ecc9c53e8220,163673dbec50af12] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:41.713 INFO 2458 --- [prometheus-minute-schedule-1,15f2ecc9c53e8220,163673dbec50af12] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:41.712313+08:00, lastProcessTime=2025-10-27T11:30:31.713467+08:00 2025-10-27 11:30:41.923 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:41.712313+08:00 2025-10-27 11:30:42.527 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:42.528 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:42.629 INFO 2458 --- [ocp-async-3,c6309f15b6c5606f,2869810134e4288a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:42.629 WARN 2458 --- [ocp-async-5,cedb49125368d260,5bd4c67153db5937] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:44.250 INFO 2458 --- [ocp-schedules-22,b15b7ea30d4e90f9,52641b8ae645cc78] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:30:44.629 WARN 2458 --- [ocp-async-3,b030bf9a0f657343,5456bbf664ac8a07] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:44.629 INFO 2458 --- [ocp-async-8,149b416ef8f68466,62b33078d354f9cb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:44.707 INFO 2458 --- [prometheus-minute-schedule-1,faa870959621d91e,de94e473cfb5993f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:44.714 INFO 2458 --- [prometheus-minute-schedule-1,faa870959621d91e,de94e473cfb5993f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:45.102 INFO 2458 --- [prometheus-second-schedule-1,75cc8b2fe251b94a,1814492c68d70272] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:30:45.381 INFO 2458 --- [ocp-schedules-23,6a9486cd739f92c8,de0bdab31ccd84b3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:45.384 INFO 2458 --- [ocp-schedules-23,6a9486cd739f92c8,de0bdab31ccd84b3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:45.528 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:45.530 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:46.629 WARN 2458 --- [ocp-async-8,d57eb5703860a2cd,b84bc1523c84a28f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:46.630 INFO 2458 --- [ocp-async-10,b302caf48741cff5,e10be48f45b43612] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:46.949 INFO 2458 --- [ocp-schedules-10,2f98e7eefb2aaa23,3bfbb2659dbcc8db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:47.605 INFO 2458 --- [ocp-schedules-24,3eefb982e20cb839,0482679cbf0b6485] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:30:47.613 INFO 2458 --- [ocp-schedules-24,3eefb982e20cb839,0482679cbf0b6485] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:47.613 INFO 2458 --- [ocp-schedules-24,3eefb982e20cb839,0482679cbf0b6485] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:30:47.707 INFO 2458 --- [prometheus-minute-schedule-1,acf7d3f97170daa4,44b97f735f3be6d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:47.713 INFO 2458 --- [prometheus-minute-schedule-1,acf7d3f97170daa4,44b97f735f3be6d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:48.530 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:48.530 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:48.538 INFO 2458 --- [ocp-async-4,f52480a5b41ce3fb,05142116d6736746] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:48.630 WARN 2458 --- [ocp-async-5,1de548b33e6b9b8c,cb028932d91f2a69] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:48.630 INFO 2458 --- [ocp-async-6,b9fd16ce75227723,f430757e85f25ccc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:49.656 INFO 2458 --- [ocp-schedules-14,38d8d4561988b0a6,2adc34f8cec3de7f] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:30:49.656+08:00, clusters=[1, 2] 2025-10-27 11:30:49.670 INFO 2458 --- [ocp-schedules-14,38d8d4561988b0a6,2adc34f8cec3de7f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:30:49.674 INFO 2458 --- [ocp-schedules-14,38d8d4561988b0a6,2adc34f8cec3de7f] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:30:50.630 WARN 2458 --- [ocp-async-6,d53ff43b15b446da,915b3e32b0c23a55] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:50.630 INFO 2458 --- [ocp-async-7,460c82d39260b8e6,81037aaf514432d0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:50.707 INFO 2458 --- [prometheus-minute-schedule-1,0698510afee565d6,0ff3bff3a1b05cdc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:50.715 INFO 2458 --- [prometheus-minute-schedule-1,0698510afee565d6,0ff3bff3a1b05cdc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:51.409 INFO 2458 --- [ocp-schedules-14,a1cad18c78886a53,3e3d05b07e0f15c4] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:51.410 INFO 2458 --- [ocp-schedules-11,63617bf4fe72ef5e,36e2f861e842ef35] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:30:51.410 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:30:51.411 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:30:51.411 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:30:51.411 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:30:51.426 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:51.427 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:51.427 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:51.427 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:51.427 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:51.428 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:51.428 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:51.429 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:51.431 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:51.431 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:51.431 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:51.431 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:51.431 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:51.432 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:51.433 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:51.433 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:51.433 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:51.435 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:51.436 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:51.436 INFO 2458 --- [ocp-schedules-24,02b3c900ff4fbeab,ddded318b3316ef2] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:30:51.437 INFO 2458 --- [host-check-clockdiff1,02b3c900ff4fbeab,a4aa5cd5123539dc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:30:51.437 INFO 2458 --- [host-check-clockdiff2,02b3c900ff4fbeab,0a8a0301dbaf86ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:30:51.437 INFO 2458 --- [host-check-clockdiff3,02b3c900ff4fbeab,da2807cd03df83c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:30:51.437 INFO 2458 --- [host-check-clockdiff4,02b3c900ff4fbeab,f6070b6e994241d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:30:51.437 INFO 2458 --- [host-check-clockdiff5,02b3c900ff4fbeab,e3879470aa194963] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:30:51.439 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:51.442 INFO 2458 --- [ocp-schedules-20,e96fc442fd6aee97,eccbdebee2d00419] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:30:51.443 INFO 2458 --- [ocp-schedules-6,d991ab45ee810584,07eae9253c0866c4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:30:51.490 INFO 2458 --- [host-check-clockdiff4,02b3c900ff4fbeab,f6070b6e994241d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535851 0 0 , error= 2025-10-27 11:30:51.493 INFO 2458 --- [host-check-clockdiff3,02b3c900ff4fbeab,da2807cd03df83c1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535851 -1 -1 , error= 2025-10-27 11:30:51.493 INFO 2458 --- [host-check-clockdiff2,02b3c900ff4fbeab,0a8a0301dbaf86ce] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535851 -1 -1 , error= 2025-10-27 11:30:51.493 INFO 2458 --- [host-check-clockdiff1,02b3c900ff4fbeab,a4aa5cd5123539dc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535851 0 0 , error= 2025-10-27 11:30:51.500 INFO 2458 --- [host-check-clockdiff5,02b3c900ff4fbeab,e3879470aa194963] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535851 0 0 , error= 2025-10-27 11:30:51.502 WARN 2458 --- [scanner-112201079-1,d991ab45ee810584,ea014fe2af203c9e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:30:51.530 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:51.531 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:30:51.535 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:30:51.535 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:30:51.535 INFO 2458 --- [host-check-agent-status5,63617bf4fe72ef5e,71e63a3d31a6a2a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:30:51.545 INFO 2458 --- [ocp-schedules-26,18fc393e2e7d5a33,3055d71bb51d271c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:30:51.547 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:51.547 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:30:51.549 INFO 2458 --- [ocp-schedules-4,826301c17ac5a540,c7ff89fbaba9980d] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:30:51.549 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:30:51.552 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:30:51.551293712+08:00 2025-10-27 11:30:51.553 INFO 2458 --- [ocp-schedules-4,826301c17ac5a540,c7ff89fbaba9980d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:51.553 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:30:51.553 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:30:51.553 INFO 2458 --- [host-check-agent-status1,63617bf4fe72ef5e,26cfe87e5ce7e911] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:30:51.553 INFO 2458 --- [ocp-schedules-9,e14939cdcf2a082f,314f0a2a03184038] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:30:51.566 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:30:51.552+08:00 2025-10-27 11:30:51.568 INFO 2458 --- [ocp-schedules-26,18fc393e2e7d5a33,3055d71bb51d271c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:51.568 INFO 2458 --- [ocp-schedules-21,06ee61b41b15eba6,484466f0ae354919] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:30:51.569 INFO 2458 --- [ocp-schedules-9,e14939cdcf2a082f,314f0a2a03184038] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:30:51.570 INFO 2458 --- [ocp-schedules-27,26484e685bb4651b,58fac5cc8484db5e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:30:51.571 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:30:51.571 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:30:51.574 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:30:51.552+08:00 and status AVAILABLE 2025-10-27 11:30:51.574 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:30:51.574 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:30:51.575 INFO 2458 --- [host-check-agent-status3,63617bf4fe72ef5e,d2924751185c749e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:30:51.590 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:30:51.590 INFO 2458 --- [ocp-schedules-27,26484e685bb4651b,58fac5cc8484db5e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:30:51.593 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:30:51.592 INFO 2458 --- [ocp-schedules-2,476983427920251a,39ba928dbca9b7e3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:30:51.595 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:30:51.596 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:30:51.595726673+08:00 2025-10-27 11:30:51.598 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:30:51.596+08:00 2025-10-27 11:30:51.599 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:30:51.599 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:30:51.599 INFO 2458 --- [host-check-agent-status4,63617bf4fe72ef5e,173f2f88729d40ad] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:30:51.599 INFO 2458 --- [ocp-schedules-21,06ee61b41b15eba6,484466f0ae354919] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:30:51.603 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:30:51.604 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:30:51.604 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:30:51.604796959+08:00 2025-10-27 11:30:51.605 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:30:51.604+08:00 2025-10-27 11:30:51.608 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:30:51.608 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:30:51.608 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:30:51.608 INFO 2458 --- [host-check-agent-status2,63617bf4fe72ef5e,25a56525e9d404c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:30:51.609 INFO 2458 --- [ocp-schedules-4,826301c17ac5a540,c7ff89fbaba9980d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:30:51.613 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:30:51.612330463+08:00 2025-10-27 11:30:51.614 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:30:51.613+08:00 2025-10-27 11:30:51.616 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:30:51.608172291+08:00 2025-10-27 11:30:51.616 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:30:51.616+08:00 2025-10-27 11:30:51.617 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:30:51.596+08:00 and status AVAILABLE 2025-10-27 11:30:51.617 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:30:51.604+08:00 and status AVAILABLE 2025-10-27 11:30:51.623 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:30:51.624 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:30:51.613+08:00 and status AVAILABLE 2025-10-27 11:30:51.625 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:30:51.630 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:30:51.616+08:00 and status AVAILABLE 2025-10-27 11:30:51.631 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:30:51.631 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:51.636 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:51.636 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:30:51.687 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:30:51.690 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:30:51.713 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:51.716 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:30:51.718 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:51.720 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:30:51.720 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:30:51.719410+08:00, lastProcessTime=2025-10-27T11:30:41.712313+08:00 2025-10-27 11:30:51.721 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:30:51.721 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:51.723 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:30:51.725 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:30:51.748 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.750 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:51.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:51.755 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:30:51.756 INFO 2458 --- [ocp-schedules-26,18fc393e2e7d5a33,3055d71bb51d271c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:30:51.760 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:51.774 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:51.792 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:30:51.792 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:30:51.799 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:51.802 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:51.813 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:30:51.819 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:30:51.819 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:51.820 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:30:51.822 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:30:51.822 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.822 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:51.830 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:30:51.830 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:30:51.831 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:30:51.848 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:30:51.854 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:30:51.859 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:51.863 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:30:51.863 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:51.873 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:30:51.882 INFO 2458 --- [ocp-schedules-7,79d48c2988246552,6b9fda5264611327] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:30:51.885 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:51.885 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:51.887 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:51.887 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.888 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:51.889 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.890 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:30:51.892 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:30:51.893 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:30:51.893 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:51.895 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:30:51.906 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:30:51.907 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:30:51.907 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:30:51.909 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:30:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:51.924 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:30:51.936 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:30:51.937 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:51.940 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:30:51.941 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:51.941 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:30:51.953 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:51.956 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:30:51.966 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:51.966 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:51.966 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.967 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:51.968 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:30:51.970 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:51.971 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:30:51.980 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:51.981 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.981 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:30:51.981 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:30:51.982 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:30:51.982 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:51.984 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:30:51.985 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:30:51.987 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:51.989 INFO 2458 --- [host-check-host-status4,a1cad18c78886a53,c87dc4b3c3989baa] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:30:51.991 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:30:51.991 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:30:51.994 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:30:51.999 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:30:52.020 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:30:51.719410+08:00 2025-10-27 11:30:52.021 INFO 2458 --- [host-check-host-status5,a1cad18c78886a53,0541b51a2d30e91b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:30:52.023 INFO 2458 --- [host-check-host-status1,a1cad18c78886a53,80b5d07f9ef7426c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:30:52.028 INFO 2458 --- [host-check-host-status2,a1cad18c78886a53,a5d39b02f6a58633] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:30:52.029 INFO 2458 --- [host-check-host-status3,a1cad18c78886a53,adc70328420cc97c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:30:52.038 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.038 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:52.071 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:30:52.083 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.083 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:52.113 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:30:52.113 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.114 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:30:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.116 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:30:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.117 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:52.133 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:30:52.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.154 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:30:52.158 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:52.159 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.159 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:30:52.195 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:30:52.195 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:30:52.197 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.197 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:30:52.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:30:52.272 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:30:52.277 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:30:52.287 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:30:52.296 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:30:52.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:30:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:30:52.421 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:30:52.421 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.422 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:30:52.423 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.424 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:30:52.461 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:30:52.501 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:30:52.502 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:30:52.503 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:30:52.540 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:30:52.540 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:30:52.540 INFO 2458 --- [ocp-schedules-9,e14939cdcf2a082f,314f0a2a03184038] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:30:52.630 WARN 2458 --- [ocp-async-10,55cfae3e49a34241,2b4781d576338995] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:52.630 INFO 2458 --- [ocp-async-4,3942eb7f398c5b1e,47da9435c4f565bb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:30:53.707 INFO 2458 --- [prometheus-minute-schedule-1,dde8ad20e6d85713,86780c7f4f3a886b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:53.714 INFO 2458 --- [prometheus-minute-schedule-1,dde8ad20e6d85713,86780c7f4f3a886b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:54.547 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:54.548 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:54.630 WARN 2458 --- [ocp-async-2,da077896a581c721,06e6bec19d8937dd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:54.630 INFO 2458 --- [ocp-async-2,605128f7feeb5f57,06ddcde57820440c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:55.392 INFO 2458 --- [ocp-schedules-10,588b0c3582202ff9,de0364515396b11f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:30:55.395 INFO 2458 --- [ocp-schedules-10,588b0c3582202ff9,de0364515396b11f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:30:55.520 INFO 2458 --- [ocp-schedules-11,551b5de39c244241,80a674f3956705d3] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:30:56.103 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,06e45e5be1594106,f690862c352d4f3a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=06e45e5be1594106, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:30:56.103 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,06e45e5be1594106,f690862c352d4f3a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=06e45e5be1594106, duration=1 ms] 2025-10-27 11:30:56.630 WARN 2458 --- [ocp-async-8,16d57220631a9e59,0138e61d9ca25f9d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:56.630 INFO 2458 --- [ocp-async-1,c4d6e0f03bab1c8f,bfbc203e87cac17f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:56.707 INFO 2458 --- [prometheus-minute-schedule-1,78caae11a3749d64,3699bc15340ea308] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:56.715 INFO 2458 --- [prometheus-minute-schedule-1,78caae11a3749d64,3699bc15340ea308] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:30:56.955 INFO 2458 --- [ocp-schedules-20,44b857cbfca8f200,493b2801fb43a84a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:30:57.548 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:30:57.549 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:30:58.538 INFO 2458 --- [ocp-async-1,dfd703ee5c917caa,29d1af3e13b25334] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:30:58.630 WARN 2458 --- [ocp-async-10,f7790080450233a7,0026742db1fc5532] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:30:58.630 INFO 2458 --- [ocp-async-4,66aad10f004f7cd4,c86a12d105be806d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:30:59.505 INFO 2458 --- [ocp-schedules-18,cc91f6fc69ddb807,28c6993d60c4c7c9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:30:59.616 INFO 2458 --- [ocp-schedules-22,979431a504cb131c,45f13cb528607722] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:30:59.621 INFO 2458 --- [ocp-schedules-22,979431a504cb131c,45f13cb528607722] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:30:59.707 INFO 2458 --- [prometheus-minute-schedule-1,8e94cf19207cad70,134d7b38d0d7aa6d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535860, interval=60 2025-10-27 11:30:59.714 INFO 2458 --- [prometheus-minute-schedule-1,8e94cf19207cad70,134d7b38d0d7aa6d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:00.550 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:00.551 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:00.631 WARN 2458 --- [ocp-async-3,28d28cbf29244855,a3eb620279617583] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:00.631 INFO 2458 --- [ocp-async-2,d909b3be366902c8,f3956d738c30ae98] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:01.714652+08:00, lastProcessTime=2025-10-27T11:30:51.719410+08:00 2025-10-27 11:31:01.760 INFO 2458 --- [ocp-schedules-29,ef5526c072420a59,43ebdf62423e3968] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:01.850 WARN 2458 --- [scanner-112201288-1,ef5526c072420a59,0274828b66b7457b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:02.098 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:01.714652+08:00 2025-10-27 11:31:02.149 INFO 2458 --- [alarm-detect-10,ef5526c072420a59,e7f84734bc12e189] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535830 ~ 1761535852 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535845, value=55.783)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535845, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535845=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:31:02.631 INFO 2458 --- [ocp-async-8,8f051b8d28166105,a41f8b5d0904a78c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:02.631 WARN 2458 --- [ocp-async-1,9e7a02d337ca39c1,2639b99d5b60a4ab] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:02.707 INFO 2458 --- [prometheus-minute-schedule-1,ebc830a710407864,f53c215478d51626] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:02.714 INFO 2458 --- [prometheus-minute-schedule-1,ebc830a710407864,f53c215478d51626] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:03.551 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:03.552 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:04.477 INFO 2458 --- [ocp-schedules-3,76a6427013964ec1,5aabdec5dcd3cf33] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:31:04.485 INFO 2458 --- [ocp-schedules-3,76a6427013964ec1,5aabdec5dcd3cf33] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:04.522 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:04.522 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:31:04.536 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:04.536 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:31:04.551 INFO 2458 --- [ocp-schedules-3,76a6427013964ec1,5aabdec5dcd3cf33] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:31:04.631 INFO 2458 --- [ocp-async-5,070c165b9cfcdc68,8bb0959fd2ced2a1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:04.631 WARN 2458 --- [ocp-async-4,f0f7e22f1c803652,a2ac539113cdca18] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:05.403 INFO 2458 --- [ocp-schedules-12,87f38b04d9ab004c,74bdbea750dbb1c4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:05.406 INFO 2458 --- [ocp-schedules-12,87f38b04d9ab004c,74bdbea750dbb1c4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:05.707 INFO 2458 --- [prometheus-minute-schedule-1,967e5e776ebdf111,2be8636be96292ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:05.714 INFO 2458 --- [prometheus-minute-schedule-1,967e5e776ebdf111,2be8636be96292ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:06.552 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:06.553 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:06.631 INFO 2458 --- [ocp-async-6,430999736cf0a313,24bc2cccd0408adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:06.631 WARN 2458 --- [ocp-async-2,b61f8d1bfac961d9,1846e39e99a6c917] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:06.961 INFO 2458 --- [ocp-schedules-24,7d00351f469cdcea,c9536cf7f7f13b2f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:07.615 INFO 2458 --- [ocp-schedules-14,e355a1909c12e3cf,c74927405fe5e12f] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:31:07.623 INFO 2458 --- [ocp-schedules-14,e355a1909c12e3cf,c74927405fe5e12f] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:07.623 INFO 2458 --- [ocp-schedules-14,e355a1909c12e3cf,c74927405fe5e12f] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:08.262 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:31:08.262+08:00, tenantSize=6 2025-10-27 11:31:08.313 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761535868262000 2025-10-27 11:31:08.313 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761535568108000 2025-10-27 11:31:08.371 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:31:08.381 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761535868376000 2025-10-27 11:31:08.381 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761535568195000 2025-10-27 11:31:08.402 INFO 2458 --- [ocp-schedules-19,c680a122fc63a79f,a766a4c6d7988d98] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:31:08.538 INFO 2458 --- [ocp-async-6,5775286e10662e96,08aefe7d55add6ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:08.631 INFO 2458 --- [ocp-async-8,02032e062a1b6b49,6714e6c053261adf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:08.631 WARN 2458 --- [ocp-async-9,d5b5f467ec39fe77,e4064da98cb15c58] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:08.707 INFO 2458 --- [prometheus-minute-schedule-1,231a32677c80b2f5,6c8b847a045b47ed] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:08.714 INFO 2458 --- [prometheus-minute-schedule-1,231a32677c80b2f5,6c8b847a045b47ed] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:09.553 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:09.554 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:10.009 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:31:10.009 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:31:10.043 INFO 2458 --- [Check obproxy1,5cebaac4ce01eafc,d1a2e104d313381e] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:31:10.044 INFO 2458 --- [Check obproxy1,5cebaac4ce01eafc,d1a2e104d313381e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:31:10.046 INFO 2458 --- [Collect obproxy1,57ed1c746c1f89c2,067168acf02f44c7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:31:10.046 INFO 2458 --- [Collect obproxy1,57ed1c746c1f89c2,067168acf02f44c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:31:10.092 INFO 2458 --- [Check obproxy1,5cebaac4ce01eafc,d1a2e104d313381e] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:31:10.096 INFO 2458 --- [Check obproxy1,5cebaac4ce01eafc,d1a2e104d313381e] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:31:10.096 INFO 2458 --- [Check obproxy1,5cebaac4ce01eafc,d1a2e104d313381e] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:31:10.097 INFO 2458 --- [Collect obproxy1,57ed1c746c1f89c2,067168acf02f44c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:31:10.100 INFO 2458 --- [Collect obproxy1,57ed1c746c1f89c2,067168acf02f44c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:31:10.103 INFO 2458 --- [Collect obproxy1,57ed1c746c1f89c2,067168acf02f44c7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:31:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:31:10.631 INFO 2458 --- [ocp-async-10,294bd5ca320ef359,28e270331b07b526] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:10.631 WARN 2458 --- [ocp-async-5,2d872ffdc1254c28,5869650309332c99] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:11.707 INFO 2458 --- [prometheus-minute-schedule-1,f18bc773b3434f07,0582f1f2d1dbc06b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:11.714 INFO 2458 --- [prometheus-minute-schedule-1,f18bc773b3434f07,0582f1f2d1dbc06b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:11.714059+08:00, lastProcessTime=2025-10-27T11:31:01.714652+08:00 2025-10-27 11:31:11.915 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:11.714059+08:00 2025-10-27 11:31:11.942 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=1000, affectRows=1000 2025-10-27 11:31:11.946 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=994, validCount=992 2025-10-27 11:31:12.002 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=1000, affectRows=1000 2025-10-27 11:31:12.007 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=999, validCount=995 2025-10-27 11:31:12.021 ERROR 2458 --- [metric-archive-second-2,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587948) Duplicate entry '835014129065861120-1761535740' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587948) Duplicate entry '835014129065861120-1761535740' for key 'PRIMARY' 2025-10-27 11:31:12.156 INFO 2458 --- [ocp-schedules-4,3672942d2c592bbb,5cd9608a2880966c] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:12.174 WARN 2458 --- [scanner-112203075-1,3672942d2c592bbb,b2a8c7ed499e091d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:12.230 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535800, nodeCount=821, affectRows=821 2025-10-27 11:31:12.283 INFO 2458 --- [alarm-detect-12,3672942d2c592bbb,2bf08411cf8b0a0f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535845 ~ 1761535862 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535860, value=58.449)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535860, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535860=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:31:12.554 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:12.554 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:12.632 WARN 2458 --- [ocp-async-6,570d13b263f541cd,f8ce7806eb318ba8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:12.632 INFO 2458 --- [ocp-async-3,96d15fce83c0c6cb,364601b7f71d542f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:12.816 INFO 2458 --- [ocp-schedules-5,0bea2b1c82f12c57,ea4bab6565ef4585] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:31:12.823 INFO 2458 --- [ocp-schedules-5,0bea2b1c82f12c57,ea4bab6565ef4585] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:12.926 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:31:12.926 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.929 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:31:12.938 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.938 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:12.964 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.964 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:12.971 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:31:12.972 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.975 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:31:12.981 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,bc7cf3b7dab76f45,7d674d668bbe13e9] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=bc7cf3b7dab76f45, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:31:12.981 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,bc7cf3b7dab76f45,7d674d668bbe13e9] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=bc7cf3b7dab76f45, duration=1 ms] 2025-10-27 11:31:12.988 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.989 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:12.991 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.991 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:12.991 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:12.991 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.017 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.017 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.019 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.020 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:13.057 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505658) Duplicate entry '835370602857963520-1761535740' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505658) Duplicate entry '835370602857963520-1761535740' for key 'PRIMARY' 2025-10-27 11:31:13.063 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.064 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.068 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.069 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:31:13.071 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.072 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.100 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.101 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.110 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.111 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.112 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.113 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.114 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.114 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.115 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.116 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.116 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.117 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=79, affectRows=79 2025-10-27 11:31:13.117 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.118 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=78, validCount=78 2025-10-27 11:31:13.118 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.119 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.120 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:31:13.122 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:31:13.129 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.130 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:31:13.132 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.133 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:31:13.162 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=1000, affectRows=1000 2025-10-27 11:31:13.165 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.165 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:31:13.166 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=996, validCount=992 2025-10-27 11:31:13.172 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:13.173 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.173 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:31:13.183 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.183 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:31:13.183 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.183 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:31:13.186 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:31:13.186 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:31:13.192 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.193 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:31:13.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.195 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:31:13.220 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.220 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:31:13.245 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.245 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:31:13.289 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.290 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.291 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.292 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.292 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.293 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.294 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.294 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.295 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.296 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.296 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.297 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.297 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.298 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.299 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.299 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.300 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.301 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.301 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.302 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:31:13.303 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.303 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:31:13.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.305 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:31:13.305 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:31:13.308 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:31:13.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:13.393 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:13.393 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:31:13.418 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:31:13.422 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:31:13.422 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:31:13.422 INFO 2458 --- [ocp-schedules-5,0bea2b1c82f12c57,ea4bab6565ef4585] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:31:14.632 INFO 2458 --- [ocp-async-9,a8b1357c8902df02,d8c0fd8cde126133] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:14.632 WARN 2458 --- [ocp-async-7,f8020847e20ce99d,1f27c918bfc10166] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:14.707 INFO 2458 --- [prometheus-minute-schedule-1,e50a7ba441e7da15,63ed3584633477b0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:14.714 INFO 2458 --- [prometheus-minute-schedule-1,e50a7ba441e7da15,63ed3584633477b0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:15.414 INFO 2458 --- [ocp-schedules-25,185fedafa05aad16,781cfa76e8e3c851] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:15.418 INFO 2458 --- [ocp-schedules-25,185fedafa05aad16,781cfa76e8e3c851] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:15.554 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:15.555 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:16.024 INFO 2458 --- [ocp-schedules-13,707dc6c0bc500c81,92af716b5dd0d20f] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:31:16.024+08:00, clusters=[1, 2] 2025-10-27 11:31:16.027 INFO 2458 --- [ocp-schedules-13,707dc6c0bc500c81,92af716b5dd0d20f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:31:16.028 INFO 2458 --- [ocp-schedules-13,707dc6c0bc500c81,92af716b5dd0d20f] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:31:16.632 INFO 2458 --- [ocp-async-8,07a109c966f16c30,c16a615a6c24f9b4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:16.632 WARN 2458 --- [ocp-async-4,a95a9f6e879245d1,9fb647eeac8618f3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:16.704 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=1000, affectRows=1000 2025-10-27 11:31:16.709 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=996, validCount=994 2025-10-27 11:31:16.967 INFO 2458 --- [ocp-schedules-3,0b179f15409fbd16,a4d04c099eae4d3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:17.707 INFO 2458 --- [prometheus-minute-schedule-1,e925ffdb719bd0f4,16be2edd53ca8473] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:17.714 INFO 2458 --- [prometheus-minute-schedule-1,e925ffdb719bd0f4,16be2edd53ca8473] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:17.846 INFO 2458 --- [ocp-schedules-29,0b6c60143462eb1a,463aa4b262390f45] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:31:17.846 INFO 2458 --- [ocp-schedules-29,0b6c60143462eb1a,463aa4b262390f45] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:31:17.897 INFO 2458 --- [Collect all tenants major compactions2,0b6c60143462eb1a,c822e77b8b49f8b0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:31:17.897 INFO 2458 --- [Collect all tenants major compactions1,0b6c60143462eb1a,4c0526cab3259166] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:31:17.897 INFO 2458 --- [Collect all tenants major compactions3,0b6c60143462eb1a,d3274e250f228f33] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:31:17.898 INFO 2458 --- [Collect all tenants major compactions4,0b6c60143462eb1a,57f027dbf0c77d0a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:31:17.898 INFO 2458 --- [Collect all tenants major compactions5,0b6c60143462eb1a,4254ed9b68ef5825] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:31:17.898 INFO 2458 --- [Collect all tenants major compactions6,0b6c60143462eb1a,08c938aa1725efd0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:31:17.932 INFO 2458 --- [Collect all tenants major compactions5,0b6c60143462eb1a,4254ed9b68ef5825] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.932 INFO 2458 --- [Collect all tenants major compactions6,0b6c60143462eb1a,08c938aa1725efd0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.933 INFO 2458 --- [Collect all tenants major compactions5,0b6c60143462eb1a,4254ed9b68ef5825] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:31:17.933 INFO 2458 --- [Collect all tenants major compactions6,0b6c60143462eb1a,08c938aa1725efd0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:31:17.937 INFO 2458 --- [Collect all tenants major compactions5,0b6c60143462eb1a,4254ed9b68ef5825] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:31:17.937 INFO 2458 --- [Collect all tenants major compactions6,0b6c60143462eb1a,08c938aa1725efd0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:31:17.948 INFO 2458 --- [Collect all tenants major compactions4,0b6c60143462eb1a,57f027dbf0c77d0a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.949 INFO 2458 --- [Collect all tenants major compactions2,0b6c60143462eb1a,c822e77b8b49f8b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.949 INFO 2458 --- [Collect all tenants major compactions3,0b6c60143462eb1a,d3274e250f228f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.949 INFO 2458 --- [Collect all tenants major compactions1,0b6c60143462eb1a,4c0526cab3259166] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:17.950 INFO 2458 --- [Collect all tenants major compactions4,0b6c60143462eb1a,57f027dbf0c77d0a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:31:17.950 INFO 2458 --- [Collect all tenants major compactions2,0b6c60143462eb1a,c822e77b8b49f8b0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:31:17.951 INFO 2458 --- [Collect all tenants major compactions3,0b6c60143462eb1a,d3274e250f228f33] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:31:17.951 INFO 2458 --- [Collect all tenants major compactions1,0b6c60143462eb1a,4c0526cab3259166] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:31:17.954 INFO 2458 --- [Collect all tenants major compactions4,0b6c60143462eb1a,57f027dbf0c77d0a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:31:17.955 INFO 2458 --- [Collect all tenants major compactions3,0b6c60143462eb1a,d3274e250f228f33] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:31:17.955 INFO 2458 --- [Collect all tenants major compactions1,0b6c60143462eb1a,4c0526cab3259166] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:31:17.956 INFO 2458 --- [Collect all tenants major compactions2,0b6c60143462eb1a,c822e77b8b49f8b0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:31:17.958 INFO 2458 --- [ocp-schedules-29,0b6c60143462eb1a,463aa4b262390f45] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:31:18.056 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535800, nodeCount=1000, affectRows=1000 2025-10-27 11:31:18.060 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535800, nodeCount=1000, validCount=986 2025-10-27 11:31:18.538 INFO 2458 --- [ocp-async-8,8870cd2b66f7b91b,5b07d764e1f6c5f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:18.555 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:18.556 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:18.632 WARN 2458 --- [ocp-async-2,8398b2353f0e906c,6b4a02a32921be6f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:18.632 INFO 2458 --- [ocp-async-3,9b1fdc3dfbcccb67,635a998d980cb314] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:18.936 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:18.944 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStand 2025-10-27 11:31:18.952 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:18.975 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:18.979 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:19.967 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:19.978 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:19.981 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.129 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.132 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.133 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.135 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.136 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.137 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.138 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.140 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.141 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.150 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.151 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.152 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.154 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.155 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.156 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.157 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.158 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.159 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:25:18.843, end=2025-10-27T11:30:18.931} 2025-10-27 11:31:20.159 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:31:20.533 INFO 2458 --- [ocp-schedules-7,d9868c280607f14f,ba59178d8b0dd5ce] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:31:20.632 WARN 2458 --- [ocp-async-1,27796b6aee5dd852,36baf784a964095b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:20.632 INFO 2458 --- [ocp-async-9,cd8383d992641507,d1740b05c143a3ea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:20.707 INFO 2458 --- [prometheus-minute-schedule-1,ba032f8824bc193d,86f9cfe2c72922c2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:20.716 INFO 2458 --- [prometheus-minute-schedule-1,ba032f8824bc193d,86f9cfe2c72922c2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:21.409 INFO 2458 --- [ocp-schedules-10,c6cd5f86914d53e6,8d910b0f4f84adb7] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:21.409 INFO 2458 --- [ocp-schedules-2,151e4451a1695ec0,2b413c0f5f85a808] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:21.409 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:31:21.409 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:31:21.409 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:31:21.410 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:31:21.410 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:31:21.410 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:31:21.410 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:31:21.410 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:31:21.411 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:31:21.411 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:31:21.422 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:21.423 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:21.423 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:21.424 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:21.424 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:21.425 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:21.426 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:21.427 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:21.427 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:21.427 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:21.428 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:21.428 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:21.428 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:21.428 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:21.428 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:21.429 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:21.429 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:21.430 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:21.430 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:21.431 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:21.433 INFO 2458 --- [ocp-schedules-20,f49c09c847041cce,becfe60da1699286] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:21.433 INFO 2458 --- [host-check-clockdiff1,f49c09c847041cce,6374cf10dcfa2986] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:31:21.434 INFO 2458 --- [host-check-clockdiff2,f49c09c847041cce,825c0c7738074a11] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:31:21.437 INFO 2458 --- [host-check-clockdiff3,f49c09c847041cce,6a349d50c0a5c89e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:31:21.437 INFO 2458 --- [host-check-clockdiff4,f49c09c847041cce,4e700470025b94a1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:31:21.437 INFO 2458 --- [host-check-clockdiff5,f49c09c847041cce,ba07e5d199fd998b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:31:21.442 INFO 2458 --- [ocp-schedules-14,9b08e22bd9c43da8,7671a8eb79c23be3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:31:21.486 INFO 2458 --- [host-check-clockdiff4,f49c09c847041cce,4e700470025b94a1] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535881 0 0 , error= 2025-10-27 11:31:21.486 INFO 2458 --- [host-check-clockdiff5,f49c09c847041cce,ba07e5d199fd998b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535881 0 0 , error= 2025-10-27 11:31:21.486 INFO 2458 --- [host-check-clockdiff1,f49c09c847041cce,6374cf10dcfa2986] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535881 0 0 , error= 2025-10-27 11:31:21.490 INFO 2458 --- [host-check-clockdiff3,f49c09c847041cce,6a349d50c0a5c89e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535881 -1 -1 , error= 2025-10-27 11:31:21.512 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:31:21.512 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:31:21.514 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:31:21.513160660+08:00 2025-10-27 11:31:21.514 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:31:21.513474808+08:00 2025-10-27 11:31:21.514 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:31:21.514+08:00 2025-10-27 11:31:21.515 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:31:21.514+08:00 2025-10-27 11:31:21.515 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:31:21.515 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:31:21.516 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:31:21.516 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:31:21.517 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:31:21.517 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:31:21.517 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:31:21.517983413+08:00 2025-10-27 11:31:21.517 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:31:21.516551883+08:00 2025-10-27 11:31:21.518 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:31:21.518 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:31:21.518+08:00 2025-10-27 11:31:21.518 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:31:21.517+08:00 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status3,c6cd5f86914d53e6,36db4dd93272502a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-agent-status2,c6cd5f86914d53e6,b8d655a889673938] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:31:21.519 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:31:21.519550345+08:00 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status4,c6cd5f86914d53e6,1a33371b27ef46a0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:31:21.520 INFO 2458 --- [host-check-agent-status5,c6cd5f86914d53e6,de0304279d0669fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:31:21.521 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:31:21.519+08:00 2025-10-27 11:31:21.522 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:31:21.523 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:31:21.523 INFO 2458 --- [host-check-agent-status1,c6cd5f86914d53e6,d89b1422e5c3db70] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:31:21.523 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:31:21.514+08:00 and status AVAILABLE 2025-10-27 11:31:21.523 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:31:21.514+08:00 and status AVAILABLE 2025-10-27 11:31:21.525 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:31:21.517+08:00 and status AVAILABLE 2025-10-27 11:31:21.526 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:31:21.518+08:00 and status AVAILABLE 2025-10-27 11:31:21.527 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:31:21.519+08:00 and status AVAILABLE 2025-10-27 11:31:21.528 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:31:21.529 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:31:21.530 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:31:21.531 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:31:21.533 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:31:21.545 INFO 2458 --- [ocp-schedules-23,4a14382767458012,c8bc84add52be023] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:31:21.546 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:21.549 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:21.549 INFO 2458 --- [ocp-schedules-3,50cd30401c046397,959a910d4363a3ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:31:21.550 INFO 2458 --- [ocp-schedules-23,4a14382767458012,c8bc84add52be023] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:21.552 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:21.553 INFO 2458 --- [ocp-schedules-3,50cd30401c046397,959a910d4363a3ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:21.553 INFO 2458 --- [ocp-schedules-3,50cd30401c046397,959a910d4363a3ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:31:21.554 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:21.554 INFO 2458 --- [ocp-schedules-19,75a8e1512ff195fa,c88018d810a52280] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:31:21.554 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:21.556 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:21.556 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:21.556 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:21.557 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:21.558 INFO 2458 --- [ocp-schedules-19,75a8e1512ff195fa,c88018d810a52280] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:21.558 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:21.559 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:21.561 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:21.565 INFO 2458 --- [ocp-schedules-18,8e5a28459e8ed2f3,7191642b87626830] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:31:21.606 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.607 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:21.608 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:31:21.628 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:21.641 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:21.643 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:31:21.643 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.643 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:21.643 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:21.648 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:31:21.648 INFO 2458 --- [ocp-schedules-23,4a14382767458012,c8bc84add52be023] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:31:21.650 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:21.660 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:21.661 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:31:21.663 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.664 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:21.666 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:31:21.680 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:31:21.689 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:31:21.697 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:31:21.705 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:31:21.709 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.709 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:21.709 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:31:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:21.715573+08:00, lastProcessTime=2025-10-27T11:31:11.714059+08:00 2025-10-27 11:31:21.718 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:31:21.725 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:21.726 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:21.726 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.727 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:31:21.727 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:31:21.735 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:21.741 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:31:21.742 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:21.746 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:21.755 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:31:21.760 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:21.763 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:31:21.764 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:31:21.779 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:31:21.789 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:31:21.792 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:21.792 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:21.793 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.793 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:31:21.793 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:21.794 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:31:21.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.797 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:31:21.799 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:21.809 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:31:21.810 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:31:21.816 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:21.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:21.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:21.831 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:31:21.833 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:21.836 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:31:21.844 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:21.844 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:21.848 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:21.854 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:31:21.862 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:31:21.866 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:31:21.867 INFO 2458 --- [host-check-host-status4,151e4451a1695ec0,772275404c2e83f4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:31:21.871 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.879 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:31:21.887 INFO 2458 --- [ocp-schedules-7,ba4d07dba8915c85,bea3dd81fca49fa2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:31:21.888 INFO 2458 --- [host-check-host-status2,151e4451a1695ec0,caee4b0e250b542d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:31:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:21.892 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.893 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:21.893 INFO 2458 --- [host-check-host-status5,151e4451a1695ec0,a45f794a15a12915] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:31:21.894 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:21.895 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:31:21.897 INFO 2458 --- [host-check-host-status3,151e4451a1695ec0,65fa67d36aad4d11] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:31:21.897 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:31:21.899 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:21.903 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:31:21.907 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:21.910 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:21.937 INFO 2458 --- [host-check-host-status1,151e4451a1695ec0,d239ad230b309c74] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:31:21.945 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:21.945 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:21.963 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:31:21.987 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:21.987 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:21.991 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:21.715573+08:00 2025-10-27 11:31:22.006 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:31:22.007 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.008 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:31:22.010 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.010 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:31:22.026 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:31:22.030 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:22.030 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:22.049 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:31:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:22.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:22.073 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:31:22.073 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535284669000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:31:22.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:22.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:22.138 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:22.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:22.179 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.179 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:22.184 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:31:22.200 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:31:22.209 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:22.212 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:22.260 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:31:22.288 INFO 2458 --- [ocp-schedules-24,b8c5ef47dbb034ef,561bd182f4d30e5e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:22.302 WARN 2458 --- [scanner-112203255-1,b8c5ef47dbb034ef,8dab1da2fe33a7cc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:22.342 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:31:22.343 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.343 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:31:22.346 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:31:22.395 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:31:22.451 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:22.452 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:22.453 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:31:22.502 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:31:22.502 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:31:22.502 INFO 2458 --- [ocp-schedules-19,75a8e1512ff195fa,c88018d810a52280] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:31:22.632 WARN 2458 --- [ocp-async-5,b1977d356668d199,5aef6fa87165b1ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:22.632 INFO 2458 --- [ocp-async-8,5efae0071bac6304,42ee2d03e080c062] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:23.707 INFO 2458 --- [prometheus-minute-schedule-1,69510703de20ba12,f21df62183f6afa1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:23.715 INFO 2458 --- [prometheus-minute-schedule-1,69510703de20ba12,f21df62183f6afa1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:23.969 INFO 2458 --- [host-check-clockdiff2,f49c09c847041cce,825c0c7738074a11] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535883 -1 -1 , error= 2025-10-27 11:31:24.557 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:24.558 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:24.632 WARN 2458 --- [ocp-async-6,19827ad60c077971,5116226ebf793b5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:24.632 INFO 2458 --- [ocp-async-2,2121a26d206c0a29,70738b33c59b1a38] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:24.668 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bff46818659fc2d6,c28ec73d1d2cc513] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&version=2&ObClusterId=1760510188, client=10.40.66.107, traceId=bff46818659fc2d6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase2,1760510188,2,{"ObRegion":"myoceanbase2","ObCluster":"myoceanbase2","ObRegionId":1760510188,"ObClusterId":1760510188,"Type":"PRIMARY","timestamp":1761535884666737,"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:31:24.674 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bff46818659fc2d6,c28ec73d1d2cc513] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase2, obClusterId=1760510188, rootServerJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, groupName=myoceanbase2, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:31:24.674 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bff46818659fc2d6,c28ec73d1d2cc513] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:31:24.674 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bff46818659fc2d6,c28ec73d1d2cc513] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase2, clusterId=1760510188, affectRows=1 2025-10-27 11:31:24.675 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bff46818659fc2d6,c28ec73d1d2cc513] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.66.107, traceId=bff46818659fc2d6, duration=8 ms] 2025-10-27 11:31:25.426 INFO 2458 --- [ocp-schedules-3,22654369b4cb1713,4ede3613f3b467fe] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:25.429 INFO 2458 --- [ocp-schedules-3,22654369b4cb1713,4ede3613f3b467fe] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:26.633 INFO 2458 --- [ocp-async-3,70a4b04fe6e28291,41c94f23c24001bf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:26.633 WARN 2458 --- [ocp-async-1,66a11d2f6474c2c0,5caa48376418bdec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:26.707 INFO 2458 --- [prometheus-minute-schedule-1,b0c85adc9ae36667,9e2308a4954a3cd2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:26.715 INFO 2458 --- [prometheus-minute-schedule-1,b0c85adc9ae36667,9e2308a4954a3cd2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:26.973 INFO 2458 --- [ocp-schedules-5,5a93eb902118bd6b,e374d540e46a9e4c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:27.558 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:27.559 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:27.626 INFO 2458 --- [ocp-schedules-13,a39d9ccf2b329f98,e12b579a6f927566] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:31:27.635 INFO 2458 --- [ocp-schedules-13,a39d9ccf2b329f98,e12b579a6f927566] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:27.635 INFO 2458 --- [ocp-schedules-13,a39d9ccf2b329f98,e12b579a6f927566] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:28.538 INFO 2458 --- [ocp-async-3,cc09183319cd143f,d885de1c8bd19875] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:28.633 WARN 2458 --- [ocp-async-5,d3a110ed6e24fb54,0c97a776a77a26fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:28.633 INFO 2458 --- [ocp-async-4,cdaf4824ac58b17a,29486fd7b6d0ca01] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:29.048 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:31:29.048 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:31:29.050 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:31:29.050 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:31:29.050 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:31:29.050 INFO 2458 --- [ocp-schedules-28,3b741371e292a58e,16d6aa1b9655e6b9] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:31:29.493 INFO 2458 --- [ocp-schedules-5,56ee83be0a0a911c,5ae6c3d8f83545ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:31:29.496 INFO 2458 --- [ocp-schedules-5,56ee83be0a0a911c,5ae6c3d8f83545ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:31:29.496 INFO 2458 --- [ocp-schedules-5,56ee83be0a0a911c,5ae6c3d8f83545ca] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:31:29.707 INFO 2458 --- [prometheus-minute-schedule-1,284490fc46db3994,3562592b59537ed2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:29.714 INFO 2458 --- [prometheus-minute-schedule-1,284490fc46db3994,3562592b59537ed2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:30.559 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:30.561 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:30.633 WARN 2458 --- [ocp-async-4,4f98fadff2c23d13,bca22cb761507185] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:30.633 INFO 2458 --- [ocp-async-10,b226bde557206994,72f2cc3633884008] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:31.713733+08:00, lastProcessTime=2025-10-27T11:31:21.715573+08:00 2025-10-27 11:31:31.924 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:31.713733+08:00 2025-10-27 11:31:32.470 INFO 2458 --- [ocp-schedules-17,9d5458da99e4b0e0,a72b1a4054fa0319] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:32.492 WARN 2458 --- [scanner-112203500-1,9d5458da99e4b0e0,9bd0e07cb35a5ddf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:32.582 INFO 2458 --- [alarm-detect-13,9d5458da99e4b0e0,90271a1abd595c2b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535860 ~ 1761535882 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535875, value=62.081)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535875, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535875=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:31:32.633 WARN 2458 --- [ocp-async-1,93eb4431233cb1c0,85a780d58f7cf96a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:32.633 INFO 2458 --- [ocp-async-2,1032e7c98176184b,cd4ab247bf7de36f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:32.707 INFO 2458 --- [prometheus-minute-schedule-1,b850c5a4b79d49f0,a2858f3b1a8be8e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:32.713 INFO 2458 --- [prometheus-minute-schedule-1,b850c5a4b79d49f0,a2858f3b1a8be8e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:33.561 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:33.562 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:33.958 INFO 2458 --- [ocp-schedules-2,62fe05be356d4871,0dfc741de1a10b7b] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:31:33.958 INFO 2458 --- [ocp-schedules-2,62fe05be356d4871,0dfc741de1a10b7b] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:31:34.633 WARN 2458 --- [ocp-async-3,66d1b9ee04825ede,8da41cf6b1ab41d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:34.633 INFO 2458 --- [ocp-async-3,d0db630f979be826,c03e43046016708b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:35.436 INFO 2458 --- [ocp-schedules-23,96eb184411c25fb0,24f39206937e974a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:35.440 INFO 2458 --- [ocp-schedules-23,96eb184411c25fb0,24f39206937e974a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:35.707 INFO 2458 --- [prometheus-minute-schedule-1,0a1570b2f6a06cb7,4c19ddd264f90adb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:35.714 INFO 2458 --- [prometheus-minute-schedule-1,0a1570b2f6a06cb7,4c19ddd264f90adb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:36.462 INFO 2458 --- [ocp-schedules-9,cefd37a36f8646ba,976b0bc8b8b4dd47] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster system event. 2025-10-27 11:31:36.474 INFO 2458 --- [ocp-schedules-9,cefd37a36f8646ba,976b0bc8b8b4dd47] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:36.562 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:36.562 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.563 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:36.566 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 02:04:47.657729, 2025-10-27 11:31:36.523, 1000] 2025-10-27 11:31:36.567 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 2, startTime 2025-10-27 02:04:47.657729, endTime 2025-10-27 11:31:36.523 2025-10-27 11:31:36.579 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.583 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091391, 2025-10-27 11:31:36.523, 1000] 2025-10-27 11:31:36.633 WARN 2458 --- [ocp-async-6,7263ebcf5f528c09,b352c152fa3847fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:36.633 INFO 2458 --- [ocp-async-4,fbe015ea282340a8,42624aa8629771e4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:36.635 INFO 2458 --- [Sync system event2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:36.640 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.645 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 12 system events to database for cluster 1 2025-10-27 11:31:36.645 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 12 system events for cluster 1, startTime 2025-10-27 08:00:44.091391, endTime 2025-10-27 11:31:36.523, limit 1000, offset 0 2025-10-27 11:31:36.672 INFO 2458 --- [Sync system event2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-17 15:45:19.719943, 2025-10-27 11:31:36.61, 1000] 2025-10-27 11:31:36.673 INFO 2458 --- [Sync system event2,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 2, startTime 2025-10-17 15:45:19.719943, endTime 2025-10-27 11:31:36.61 2025-10-27 11:31:36.684 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:36.685 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.688 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, rs_svr_ip, rs_svr_port FROM DBA_OB_ROOTSERVICE_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( not (module = "balancer" AND event IN ("start_sql_bkgd_task","finish_sql_bkgd_task")) AND not (module = "schema_recycler")) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2025-10-27 08:00:44.091391, 2025-10-27 11:31:36.523, 1000, 1000] 2025-10-27 11:31:36.696 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync system events: cluster 1, startTime 2025-10-27 08:00:44.091391, endTime 2025-10-27 11:31:36.523 2025-10-27 11:31:36.753 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:36.754 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.775 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit, args: [2025-10-27 08:00:44.091611, 2025-10-27 11:31:36.709, 1000] 2025-10-27 11:31:36.777 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Sync-ed 2 server events for cluster 1, startTime 2025-10-27 08:00:44.091611, endTime 2025-10-27 11:31:36.709, limit 1000, offset 0 2025-10-27 11:31:36.796 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Batch saved 2 server events to database for cluster 1 2025-10-27 11:31:36.839 INFO 2458 --- [Sync system event1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:36.840 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:36.862 INFO 2458 --- [Sync system event1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ timestamp as gmt_create, module, event, name1, value1, name2, value2, name3, value3, name4, value4, name5, value5, name6, value6, extra_info, svr_ip, svr_port FROM DBA_OB_SERVER_EVENT_HISTORY WHERE 1=1 AND timestamp > :startTime AND timestamp <= :endTime AND ( (module = "sql") OR (module = "daily_merge") OR (module = "election") OR (module = "physical_restore") OR (module = "observice") OR (module = "observer") OR (module = "clog_restore") OR (module = "warmup") OR (module = "log_archive") OR (module = "archive_log") OR (module = "minor_merge") OR (module = "ddl" AND event IN ("create_partition","batch_create_partition","add_replica","remove_member","batch_remove_member","batch_modify_quorum","batch_change_replica","migrate_replica","remove_replica","rebuild_replica")) OR (module = "freeze" AND event IN ("do minor freeze fail")) OR (module = "storage" AND event IN ("add_replica failed","migrate_replica failed","reubild_replica failed","change_replica failed","log is new enough, turn off rebuild flag","remove_replica failed","restore follower replica failed","restore_replica failed","batch change_replica failed","copy global index failed","restore_standby_replica failed","rebuild_replica failed"))) ORDER BY timestamp ASC LIMIT :limit OFFSET :offset, args: [2025-10-27 08:00:44.091611, 2025-10-27 11:31:36.709, 1000, 1000] 2025-10-27 11:31:36.872 INFO 2458 --- [Sync system event1,,] .o.o.o.i.c.ClusterSystemEventServiceImpl : Finished sync server events: cluster 1, startTime 2025-10-27 08:00:44.091611, endTime 2025-10-27 11:31:36.709 2025-10-27 11:31:36.872 INFO 2458 --- [ocp-schedules-9,cefd37a36f8646ba,976b0bc8b8b4dd47] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync system event executed completed, allCount=2, successCount=2 2025-10-27 11:31:36.979 INFO 2458 --- [ocp-schedules-13,d0a56d245152f4ed,6760f0d7db0a1e87] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:37.708 INFO 2458 --- [prometheus-cache-monitor1,e0c1ca91135da09a,894a8cd8d3ac5190] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20349,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5079,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:31:37.708 INFO 2458 --- [prometheus-cache-monitor1,e0c1ca91135da09a,894a8cd8d3ac5190] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:31:38.538 INFO 2458 --- [ocp-async-4,749d6ec819d953c8,9c4f1777f2ce0375] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:38.633 INFO 2458 --- [ocp-async-7,eeadd0f5363030a2,be290aa871fa4d52] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:38.633 WARN 2458 --- [ocp-async-2,8fcf0dd8014bfa6d,3fab5ba9aa35708b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:38.707 INFO 2458 --- [prometheus-minute-schedule-1,45a2b0c6ef087d1f,91c9b609fa8015e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:38.714 INFO 2458 --- [prometheus-minute-schedule-1,45a2b0c6ef087d1f,91c9b609fa8015e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:38.817 INFO 2458 --- [ocp-schedules-18,f6ff72a80d4c74c4,23876a86884313d1] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:31:38.824 INFO 2458 --- [ocp-schedules-18,f6ff72a80d4c74c4,23876a86884313d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:38.824 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:31:38.824 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:31:38.825 INFO 2458 --- [ocp-schedules-18,f6ff72a80d4c74c4,23876a86884313d1] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:31:39.563 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:39.564 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:39.759 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:39.759 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:31:39.759 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:31:39.759 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:31:39.759 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:31:39.759 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:31:39.769 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:39.772 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:39.772 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:39.772 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:39.773 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:39.773 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:39.775 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:39.776 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:39.776 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:39.777 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:39.796 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:31:39.796 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:31:39.827 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:31:39.831 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:31:39.832 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:31:39.833 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:31:39.833 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:31:39.835 INFO 2458 --- [ocp-schedules-20,739235e55d1cebf6,fa586a5edec554b0] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:31:39.899 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:39.900 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:31:39.912 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:39.913 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:31:39.969 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:39.971 INFO 2458 --- [ocp-schedules-29,af9a13ecf0e31bd5,76bb8d4cb53d3c65] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:31:40.006 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:31:40.008 INFO 2458 --- [ocp-schedules-23,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:31:40.037 INFO 2458 --- [Check obproxy1,19fd9ef23ccd1ce1,065bec26d6243210] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:31:40.037 INFO 2458 --- [Check obproxy1,19fd9ef23ccd1ce1,065bec26d6243210] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:31:40.049 INFO 2458 --- [Collect obproxy1,2734b1ef50a747c1,520e390bd4909c06] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:31:40.049 INFO 2458 --- [Collect obproxy1,2734b1ef50a747c1,520e390bd4909c06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:31:40.071 INFO 2458 --- [Check obproxy1,19fd9ef23ccd1ce1,065bec26d6243210] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:31:40.074 INFO 2458 --- [Check obproxy1,19fd9ef23ccd1ce1,065bec26d6243210] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:31:40.074 INFO 2458 --- [Check obproxy1,19fd9ef23ccd1ce1,065bec26d6243210] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:31:40.080 INFO 2458 --- [Collect obproxy1,2734b1ef50a747c1,520e390bd4909c06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:31:40.083 INFO 2458 --- [Collect obproxy1,2734b1ef50a747c1,520e390bd4909c06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:31:40.086 INFO 2458 --- [Collect obproxy1,2734b1ef50a747c1,520e390bd4909c06] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:31:40.634 WARN 2458 --- [ocp-async-3,8f86e16b46227dc7,839ce37e0e6b73b7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:40.634 INFO 2458 --- [ocp-async-5,7939d2a95b5c9a98,faa1fd856985d848] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:41.278 INFO 2458 --- [ocp-schedules-13,5358c405a4d321c5,0f2c7d3a4f4fbe8f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:31:41.707 INFO 2458 --- [prometheus-minute-schedule-1,f629743e8c71b43e,2c98c00c48ad0039] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:41.714 INFO 2458 --- [prometheus-minute-schedule-1,f629743e8c71b43e,2c98c00c48ad0039] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:41.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:41.721214+08:00, lastProcessTime=2025-10-27T11:31:31.713733+08:00 2025-10-27 11:31:41.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:41.721214+08:00 2025-10-27 11:31:42.565 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:42.565 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:42.588 INFO 2458 --- [ocp-schedules-17,8707e54fa8e23527,6f5ec17071572cc7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:42.619 WARN 2458 --- [scanner-112203707-1,8707e54fa8e23527,21dd695ca6941ca3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:42.634 INFO 2458 --- [ocp-async-6,ec5831d3a391b703,5a45646ff2b4330f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:42.634 WARN 2458 --- [ocp-async-9,da5162f3078195d7,ab6b5901932c88a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:42.771 INFO 2458 --- [alarm-detect-1,8707e54fa8e23527,7057db717f9d92f1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535875 ~ 1761535892 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535890, value=58.355)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535890, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535890=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:31:44.262 INFO 2458 --- [ocp-schedules-25,132919627cb0bea2,d9bf4d232090031d] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:31:44.634 INFO 2458 --- [ocp-async-10,325ab8129f5b3f4c,3c71011c5b832413] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:44.634 WARN 2458 --- [ocp-async-2,1b00bcba14ae17ff,cee96d94a8075a2a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:44.707 INFO 2458 --- [prometheus-minute-schedule-1,21963d32bc7c8fa1,69beebff65dd1a9b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:44.714 INFO 2458 --- [prometheus-minute-schedule-1,21963d32bc7c8fa1,69beebff65dd1a9b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:45.102 INFO 2458 --- [prometheus-second-schedule-1,a90536447c662e2b,8b4682f7ffafbb58] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:31:45.448 INFO 2458 --- [ocp-schedules-4,0d4d0cd611dd591b,fc86d38f9d23f196] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:45.452 INFO 2458 --- [ocp-schedules-4,0d4d0cd611dd591b,fc86d38f9d23f196] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:45.565 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:45.566 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:46.634 INFO 2458 --- [ocp-async-7,59e7a63eb88c221e,0de327dd97f42aea] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:46.634 WARN 2458 --- [ocp-async-3,eca26ed7df788c0a,62150bd054f1cd2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:46.986 INFO 2458 --- [ocp-schedules-29,36268ac702dd6882,e2dfe9d1c2bd4eef] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:47.637 INFO 2458 --- [ocp-schedules-15,97b73f8db97ae788,af2b59430d47a445] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:31:47.649 INFO 2458 --- [ocp-schedules-15,97b73f8db97ae788,af2b59430d47a445] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:47.649 INFO 2458 --- [ocp-schedules-15,97b73f8db97ae788,af2b59430d47a445] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:31:47.707 INFO 2458 --- [prometheus-minute-schedule-1,8e1213546701a34e,6d9158c3dbf1cf5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:47.714 INFO 2458 --- [prometheus-minute-schedule-1,8e1213546701a34e,6d9158c3dbf1cf5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:48.539 INFO 2458 --- [ocp-async-7,4e71a919bf71be6c,bfd79b13c9b7562b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:48.566 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:48.567 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:48.634 INFO 2458 --- [ocp-async-4,a4e61ed7d21ffbb1,b9a087ecdb940831] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:48.634 WARN 2458 --- [ocp-async-6,108245e140efc8b1,ae09f3a1b05c6c52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:49.726 INFO 2458 --- [ocp-schedules-2,6931b74acf088366,0177fffd01782ebc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:31:49.726+08:00, clusters=[1, 2] 2025-10-27 11:31:49.747 INFO 2458 --- [ocp-schedules-2,6931b74acf088366,0177fffd01782ebc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:31:49.750 INFO 2458 --- [ocp-schedules-2,6931b74acf088366,0177fffd01782ebc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:31:50.634 INFO 2458 --- [ocp-async-2,a87d6de964071ffb,cd73330c4ba751b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:50.634 WARN 2458 --- [ocp-async-10,768ce1abcd854488,aae0c1dea9569409] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:50.707 INFO 2458 --- [prometheus-minute-schedule-1,1804ef2274caed1e,57e95b77ae88e05c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:50.715 INFO 2458 --- [prometheus-minute-schedule-1,1804ef2274caed1e,57e95b77ae88e05c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:51.408 INFO 2458 --- [ocp-schedules-10,789e483a5c375437,5af9814f4b4a418c] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:51.408 INFO 2458 --- [ocp-schedules-27,dd7edc40b35d0b88,9a1c7d1a82c690d6] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:51.409 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:31:51.409 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:31:51.409 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:31:51.409 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:31:51.409 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:31:51.410 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:31:51.410 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:31:51.410 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:31:51.410 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:31:51.411 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:31:51.422 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:51.422 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:51.423 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:51.423 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:51.426 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:51.426 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:51.427 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:51.428 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:51.428 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:51.429 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:51.429 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:51.430 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:51.430 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:51.431 INFO 2458 --- [ocp-schedules-20,ff5e4c5981622a99,f49f48a863fc9794] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:31:51.431 INFO 2458 --- [host-check-clockdiff1,ff5e4c5981622a99,b94a10d5459560ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:31:51.431 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:51.431 INFO 2458 --- [host-check-clockdiff2,ff5e4c5981622a99,06103a141b291c0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:31:51.431 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:51.432 INFO 2458 --- [host-check-clockdiff4,ff5e4c5981622a99,cb9f95a49bf3a18c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:31:51.433 INFO 2458 --- [host-check-clockdiff3,ff5e4c5981622a99,a7be67699c193a74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:31:51.432 INFO 2458 --- [host-check-clockdiff5,ff5e4c5981622a99,c4ef308d2268adc6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:31:51.446 INFO 2458 --- [host-check-clockdiff2,ff5e4c5981622a99,06103a141b291c0e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535911 -1 -1 , error= 2025-10-27 11:31:51.446 INFO 2458 --- [ocp-schedules-3,c7c2ea8582b49f21,994814d969227f9b] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:31:51.458 INFO 2458 --- [host-check-clockdiff1,ff5e4c5981622a99,b94a10d5459560ae] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535911 0 0 , error= 2025-10-27 11:31:51.461 INFO 2458 --- [host-check-clockdiff5,ff5e4c5981622a99,c4ef308d2268adc6] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535911 1 1 , error= 2025-10-27 11:31:51.467 INFO 2458 --- [host-check-clockdiff4,ff5e4c5981622a99,cb9f95a49bf3a18c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535911 -1 -1 , error= 2025-10-27 11:31:51.467 INFO 2458 --- [host-check-clockdiff3,ff5e4c5981622a99,a7be67699c193a74] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535911 -1 -1 , error= 2025-10-27 11:31:51.525 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:31:51.529 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:31:51.529 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:31:51.529 INFO 2458 --- [host-check-agent-status4,dd7edc40b35d0b88,d07dffcda5b3ace1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:31:51.530 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:31:51.531 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:31:51.532 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:31:51.532444600+08:00 2025-10-27 11:31:51.533 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:31:51.532+08:00 2025-10-27 11:31:51.533 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:31:51.533 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:31:51.534 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:31:51.534 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:31:51.534 INFO 2458 --- [host-check-agent-status5,dd7edc40b35d0b88,802b4b83133e186f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:31:51.534 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:31:51.535 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:31:51.534095756+08:00 2025-10-27 11:31:51.535 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:31:51.535 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:31:51.535+08:00 2025-10-27 11:31:51.538 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:31:51.538 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:31:51.538 INFO 2458 --- [host-check-agent-status2,dd7edc40b35d0b88,0920051016fda7fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:31:51.538 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:31:51.538461086+08:00 2025-10-27 11:31:51.538 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:31:51.535889713+08:00 2025-10-27 11:31:51.539 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:31:51.538+08:00 2025-10-27 11:31:51.539 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:31:51.539 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:31:51.538+08:00 2025-10-27 11:31:51.540 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:31:51.539603102+08:00 2025-10-27 11:31:51.540 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:31:51.540 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:31:51.540+08:00 2025-10-27 11:31:51.541 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:31:51.532+08:00 and status AVAILABLE 2025-10-27 11:31:51.541 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:31:51.535+08:00 and status AVAILABLE 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status1,dd7edc40b35d0b88,9fef86e027230ccf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:31:51.544 INFO 2458 --- [host-check-agent-status3,dd7edc40b35d0b88,c9661f3ceab3dd79] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:31:51.545 INFO 2458 --- [ocp-schedules-22,23fcc8838b4b7cd7,1b3af620fd4b71e3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:31:51.546 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:31:51.548 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:31:51.538+08:00 and status AVAILABLE 2025-10-27 11:31:51.549 INFO 2458 --- [ocp-schedules-13,9640aacfecebf0e1,652e84a99ffe39f9] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:31:51.549 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:31:51.549 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:31:51.540+08:00 and status AVAILABLE 2025-10-27 11:31:51.549 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:31:51.538+08:00 and status AVAILABLE 2025-10-27 11:31:51.550 INFO 2458 --- [ocp-schedules-22,23fcc8838b4b7cd7,1b3af620fd4b71e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:51.552 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:31:51.553 INFO 2458 --- [ocp-schedules-13,9640aacfecebf0e1,652e84a99ffe39f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:51.553 INFO 2458 --- [ocp-schedules-13,9640aacfecebf0e1,652e84a99ffe39f9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:31:51.553 INFO 2458 --- [ocp-schedules-13,f32c0e34d05c7e1c,ef3002516ab82b11] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:31:51.555 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:31:51.555 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:31:51.557 INFO 2458 --- [ocp-schedules-13,f32c0e34d05c7e1c,ef3002516ab82b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:51.566 INFO 2458 --- [ocp-schedules-6,6135c6963a24387c,ef230b715f94b253] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:31:51.567 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:51.568 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:51.568 INFO 2458 --- [ocp-schedules-5,d1675d22472d76bb,82933751a8f162cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:31:51.569 INFO 2458 --- [ocp-schedules-12,12a40d54cf62892a,bc002698b8bb61f8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:31:51.570 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:51.570 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:31:51.573 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:31:51.573 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:51.577 INFO 2458 --- [ocp-schedules-12,12a40d54cf62892a,bc002698b8bb61f8] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:31:51.580 INFO 2458 --- [ocp-schedules-5,d1675d22472d76bb,82933751a8f162cc] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:31:51.580 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:51.584 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:51.586 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:31:51.588 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:31:51.593 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.594 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:51.595 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:31:51.596 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:31:51.599 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:31:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:51.626 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:31:51.627 INFO 2458 --- [ocp-schedules-22,23fcc8838b4b7cd7,1b3af620fd4b71e3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:31:51.627 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.627 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:51.633 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:51.635 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:51.646 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:51.652 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:51.657 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:31:51.662 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:31:51.662 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:31:51.663 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:31:51.667 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:31:51.678 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.679 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:51.682 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:51.682 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.683 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:31:51.683 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:31:51.695 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:31:51.697 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:31:51.697 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:31:51.699 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:31:51.711 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:31:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:31:51.714965+08:00, lastProcessTime=2025-10-27T11:31:41.721214+08:00 2025-10-27 11:31:51.722 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:51.726 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:51.729 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:51.739 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:51.744 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:31:51.746 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:31:51.747 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:31:51.754 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:31:51.755 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:51.755 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:51.756 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.756 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:51.756 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.757 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:31:51.757 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.761 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:31:51.761 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:51.764 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:31:51.767 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:31:51.775 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:31:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:51.782 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:31:51.783 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:31:51.794 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:31:51.802 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:31:51.808 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:51.809 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:51.815 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:51.829 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:31:51.829 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:51.829 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:31:51.833 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:51.838 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:31:51.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:51.840 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.841 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:51.842 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:31:51.850 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:31:51.850 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:31:51.851 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:31:51.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:31:51.867 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:31:51.867 INFO 2458 --- [host-check-host-status2,789e483a5c375437,0ddfecc33967d6b8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:31:51.868 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:31:51.869 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.870 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:31:51.872 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.875 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:31:51.877 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:51.880 INFO 2458 --- [host-check-host-status4,789e483a5c375437,18b2d7257880a7f6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:31:51.887 INFO 2458 --- [host-check-host-status5,789e483a5c375437,55f00e84bbef50f9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:31:51.887 INFO 2458 --- [host-check-host-status3,789e483a5c375437,bf2684dd1914a27b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:31:51.890 INFO 2458 --- [ocp-schedules-7,5797bdd5c953ba06,d2437fbe18c3a563] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:31:51.907 INFO 2458 --- [host-check-host-status1,789e483a5c375437,59cc08aa44d439d5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:31:51.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:51.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:51.937 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:31:51.956 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:31:51.714965+08:00 2025-10-27 11:31:51.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:51.965 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:51.981 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:31:51.981 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.982 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:31:51.983 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:51.984 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:31:51.997 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:51.997 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:51.999 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:31:52.022 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:52.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:31:52.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:52.032 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:31:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:31:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:31:52.068 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:52.068 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:31:52.103 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:31:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:52.142 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:31:52.145 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:31:52.155 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:31:52.163 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:31:52.167 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:31:52.211 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:31:52.272 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:31:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:31:52.275 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:52.276 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:31:52.320 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:31:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:31:52.360 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:31:52.361 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:31:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:31:52.399 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:31:52.400 INFO 2458 --- [ocp-schedules-13,f32c0e34d05c7e1c,ef3002516ab82b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:31:52.635 WARN 2458 --- [ocp-async-5,28a6a8b79ed870cf,730568b05b6c775b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:52.635 INFO 2458 --- [ocp-async-6,c5a6ac91ad1ab191,2fc72a80569da694] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:52.780 INFO 2458 --- [ocp-schedules-16,5f8d3e6b15d4485c,d37b3cb13a092c67] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:31:52.820 WARN 2458 --- [scanner-112203951-1,5f8d3e6b15d4485c,bb716c9bc180db1d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:31:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:31:53.236 INFO 2458 --- [ocp-schedules-19,6e734c753bc4ae4b,4e07dd7dfe88def1] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20349,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5079,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:31:53.236 INFO 2458 --- [ocp-schedules-19,6e734c753bc4ae4b,4e07dd7dfe88def1] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:31:53.707 INFO 2458 --- [prometheus-minute-schedule-1,f0c87db1626211eb,004b20b959c25301] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:53.714 INFO 2458 --- [prometheus-minute-schedule-1,f0c87db1626211eb,004b20b959c25301] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:54.463 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,7142048386ffd590,a0bd08ea472f3437] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=7142048386ffd590, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:31:54.463 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,7142048386ffd590,a0bd08ea472f3437] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=7142048386ffd590, duration=0 ms] 2025-10-27 11:31:54.465 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b69d6ef45d0907d4,35f4f258387fd46c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest, client=10.40.67.22, traceId=b69d6ef45d0907d4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,false,] 2025-10-27 11:31:54.465 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,b69d6ef45d0907d4,35f4f258387fd46c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b69d6ef45d0907d4, duration=1 ms] 2025-10-27 11:31:54.568 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:54.569 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:54.635 WARN 2458 --- [ocp-async-9,4c62a6d40983cea3,7669cac03e3326fe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:54.635 INFO 2458 --- [ocp-async-4,a01dccfff0574df4,dd34b5712df025ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:55.469 INFO 2458 --- [ocp-schedules-9,d137f8a8075c134e,b93542896a8d4656] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:31:55.473 INFO 2458 --- [ocp-schedules-9,d137f8a8075c134e,b93542896a8d4656] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:31:55.567 INFO 2458 --- [ocp-schedules-17,ac323aa077efdfa2,4b3a1675cd33248b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:31:56.635 INFO 2458 --- [ocp-async-1,6c5d76dab4c7470b,56445132ab232bc0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:56.635 WARN 2458 --- [ocp-async-2,0721025c846ae1ff,bbf919d2c409cb85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:56.707 INFO 2458 --- [prometheus-minute-schedule-1,37b1a874b585cc62,9b08bc9320259e4f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:56.714 INFO 2458 --- [prometheus-minute-schedule-1,37b1a874b585cc62,9b08bc9320259e4f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:31:56.992 INFO 2458 --- [ocp-schedules-9,381ce958203072a2,2b9a7f76ec13d757] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:31:57.569 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:31:57.570 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:31:58.352 INFO 2458 --- [ocp-schedules-12,24e9a3983f57a4e7,a05cd91602b1bab0] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:31:58.360 INFO 2458 --- [ocp-schedules-12,24e9a3983f57a4e7,a05cd91602b1bab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:31:58.400 INFO 2458 --- [ocp-schedules-12,24e9a3983f57a4e7,a05cd91602b1bab0] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:31:58.539 INFO 2458 --- [ocp-async-1,bf3e178ee495573b,02b6e57ea27391b1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:31:58.635 WARN 2458 --- [ocp-async-7,27003808d97f68d6,ca7518a7ac38daf7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:31:58.635 INFO 2458 --- [ocp-async-6,3335e66b28bb9da4,632720e19b67118f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:31:59.516 INFO 2458 --- [ocp-schedules-6,52935da2c1e416df,8fcd1e5c3a0d2494] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:31:59.707 INFO 2458 --- [prometheus-minute-schedule-1,cb6ba96ffd01ed27,c5014e10aa116e1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535920, interval=60 2025-10-27 11:31:59.714 INFO 2458 --- [prometheus-minute-schedule-1,cb6ba96ffd01ed27,c5014e10aa116e1f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:00.570 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:00.570 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:00.635 WARN 2458 --- [ocp-async-9,0f497053b0ecc037,07f62c92d19d4309] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:00.635 INFO 2458 --- [ocp-async-9,2ede0c310a8c1385,5093f324053d5370] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:01.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:01.718107+08:00, lastProcessTime=2025-10-27T11:31:51.714965+08:00 2025-10-27 11:32:01.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:01.718107+08:00 2025-10-27 11:32:02.635 WARN 2458 --- [ocp-async-2,f1cca5f78a9f245b,a66eaef74994366c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:02.635 INFO 2458 --- [ocp-async-1,3650e38f98fe0ce5,29e1b96727443c1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:02.707 INFO 2458 --- [prometheus-minute-schedule-1,3ef3f38c89d02aef,91e066a462f361bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:02.713 INFO 2458 --- [prometheus-minute-schedule-1,3ef3f38c89d02aef,91e066a462f361bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:02.900 INFO 2458 --- [ocp-schedules-14,8a42ee2377ca7720,a1416b4bc4e0138f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:03.108 WARN 2458 --- [scanner-112204136-1,8a42ee2377ca7720,6af13ae177fb62bb] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:03.570 INFO 2458 --- [alarm-detect-8,8a42ee2377ca7720,ad7dd8ab8edbfc7a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535890 ~ 1761535913 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535905, value=44.977)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535905, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535905=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:32:03.571 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:03.571 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:04.551 INFO 2458 --- [ocp-schedules-27,f300381ec5482cd6,27486e2351b8f9ad] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:32:04.560 INFO 2458 --- [ocp-schedules-27,f300381ec5482cd6,27486e2351b8f9ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:04.597 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:04.598 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:32:04.609 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:04.609 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:32:04.624 INFO 2458 --- [ocp-schedules-27,f300381ec5482cd6,27486e2351b8f9ad] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:32:04.635 WARN 2458 --- [ocp-async-7,5be6aa3d8829ce80,7f780434fa15c4ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:04.636 INFO 2458 --- [ocp-async-6,c0f131fa6ec4070f,bd18ad483f3dc37a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:05.481 INFO 2458 --- [ocp-schedules-22,2ca04b386bec48c1,8bcb7757e595ad47] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:05.484 INFO 2458 --- [ocp-schedules-22,2ca04b386bec48c1,8bcb7757e595ad47] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:05.707 INFO 2458 --- [prometheus-minute-schedule-1,d952faf1e2691ead,fa6e466420d6ffc1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:05.714 INFO 2458 --- [prometheus-minute-schedule-1,d952faf1e2691ead,fa6e466420d6ffc1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:32:06.049 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:32:06.089 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:32:06.115 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:32:06.115 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:32:06.115 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:32:06.115 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-27T11:29:30, target=2025-10-27T11:27:30, granule=120000ms, delay=120000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:27:30, target=2025-10-27T11:22:30, granule=600000ms, delay=300000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:32:06.118 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=120ms, period=120000ms 2025-10-27 11:32:06.571 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:06.572 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:06.636 WARN 2458 --- [ocp-async-6,e2489ae6aa7bf673,96d8c992721da5b9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:06.636 INFO 2458 --- [ocp-async-6,99337c8d6cd958c0,737eb61fe587ab8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:06.998 INFO 2458 --- [ocp-schedules-4,f3739b9bfa59b949,b13d69b594070fd7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:07.652 INFO 2458 --- [ocp-schedules-17,f874c6ec5f422e2c,cc815b23c328f6dc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:32:07.660 INFO 2458 --- [ocp-schedules-17,f874c6ec5f422e2c,cc815b23c328f6dc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:07.660 INFO 2458 --- [ocp-schedules-17,f874c6ec5f422e2c,cc815b23c328f6dc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:08.539 INFO 2458 --- [ocp-async-4,bbad5dae6f0f455a,49be8ea4fbd3b5ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:08.636 INFO 2458 --- [ocp-async-2,ec35452ea1b46fcd,12cc9f2e5ab7538c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:08.636 WARN 2458 --- [ocp-async-8,f0be2cf208ae937e,a1ff3c1e84be8fd2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:08.707 INFO 2458 --- [prometheus-minute-schedule-1,15c501bcd53a9da8,5bcb3b7feeec7f26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:08.714 INFO 2458 --- [prometheus-minute-schedule-1,15c501bcd53a9da8,5bcb3b7feeec7f26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:09.572 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:09.573 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:10.008 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:32:10.009 INFO 2458 --- [ocp-schedules-4,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:32:10.042 INFO 2458 --- [Check obproxy1,c2cf550a2fc66fd4,620895f84d483fb5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:32:10.042 INFO 2458 --- [Check obproxy1,c2cf550a2fc66fd4,620895f84d483fb5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:32:10.044 INFO 2458 --- [Collect obproxy1,540357246798766c,d891376a699a05fb] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:32:10.044 INFO 2458 --- [Collect obproxy1,540357246798766c,d891376a699a05fb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:32:10.073 INFO 2458 --- [Check obproxy1,c2cf550a2fc66fd4,620895f84d483fb5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:32:10.074 INFO 2458 --- [Collect obproxy1,540357246798766c,d891376a699a05fb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:32:10.077 INFO 2458 --- [Check obproxy1,c2cf550a2fc66fd4,620895f84d483fb5] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:32:10.077 INFO 2458 --- [Check obproxy1,c2cf550a2fc66fd4,620895f84d483fb5] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:32:10.078 INFO 2458 --- [Collect obproxy1,540357246798766c,d891376a699a05fb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:32:10.081 INFO 2458 --- [Collect obproxy1,540357246798766c,d891376a699a05fb] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:32:10.187 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:32:10.636 INFO 2458 --- [ocp-async-2,8a119d3c9f705910,b697093dcfe47996] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:10.636 WARN 2458 --- [ocp-async-5,6e3c5633a8630c53,72ed19e893f7f957] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:11.344 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,6d26e990db5ac736,6fe118107b136a3c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=6d26e990db5ac736, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:32:11.345 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,6d26e990db5ac736,6fe118107b136a3c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=6d26e990db5ac736, duration=1 ms] 2025-10-27 11:32:11.476 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=1000, affectRows=1000 2025-10-27 11:32:11.480 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=994, validCount=992 2025-10-27 11:32:11.707 INFO 2458 --- [prometheus-minute-schedule-1,9b7b8f41018f744d,e62755946c1c0858] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:11.714 INFO 2458 --- [prometheus-minute-schedule-1,9b7b8f41018f744d,e62755946c1c0858] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:11.723 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:11.722027+08:00, lastProcessTime=2025-10-27T11:32:01.718107+08:00 2025-10-27 11:32:11.935 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:11.722027+08:00 2025-10-27 11:32:12.467 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505658) Duplicate entry '835014129065861120-1761535800' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505658) Duplicate entry '835014129065861120-1761535800' for key 'PRIMARY' 2025-10-27 11:32:12.573 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:12.573 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:12.636 INFO 2458 --- [ocp-async-2,79e6110f570eb6ad,91b5e09e6b962503] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:12.636 WARN 2458 --- [ocp-async-6,61581f73952d11ae,891fd166f549c6b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:13.422 INFO 2458 --- [ocp-schedules-22,338940db69abd4d3,c570431829bd4d0a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:32:13.429 INFO 2458 --- [ocp-schedules-22,338940db69abd4d3,c570431829bd4d0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:13.529 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:32:13.530 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.533 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:32:13.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.570 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.570 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.576 INFO 2458 --- [ocp-schedules-16,63bba0a785c631a1,39274c30630842ac] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:13.600 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.601 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.603 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:32:13.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.603 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.608 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:32:13.625 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.626 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.630 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.631 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.656 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.657 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.659 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.691 WARN 2458 --- [scanner-112205950-1,63bba0a785c631a1,95a876558bba9430] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:13.692 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.714 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.714 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:32:13.717 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.717 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.720 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.721 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:13.722 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.723 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:13.723 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.724 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:13.725 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.726 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:13.727 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.728 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:13.728 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.729 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:13.730 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:32:13.733 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:32:13.751 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.751 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.791 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:13.792 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.792 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.792 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:32:13.793 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:32:13.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.795 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:32:13.802 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:13.802 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:32:13.802 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:13.803 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:32:13.806 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:32:13.830 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.831 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:32:13.806 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:32:13.872 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.872 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:32:13.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.875 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:32:13.891 INFO 2458 --- [alarm-detect-10,63bba0a785c631a1,21f62d5e382ef4fd] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535905 ~ 1761535923 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535920, value=85.961)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535920, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535920=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:32:13.907 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.908 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:32:13.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.934 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:32:13.986 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.987 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:13.989 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.990 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:13.990 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.991 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:13.992 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.993 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:13.993 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.994 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:13.995 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.996 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:13.996 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.997 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:13.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:13.998 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:13.999 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:14.000 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:14.000 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:14.001 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:32:14.002 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:14.002 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:32:14.003 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:14.004 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:32:14.004 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:32:14.007 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:32:14.088 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:14.089 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:14.089 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:32:14.117 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:32:14.122 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:32:14.122 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:32:14.122 INFO 2458 --- [ocp-schedules-22,338940db69abd4d3,c570431829bd4d0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:32:14.636 WARN 2458 --- [ocp-async-2,8ca9689b9db0a92c,9dda08cfd272ea1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:14.636 INFO 2458 --- [ocp-async-8,3895029d46019d3e,bfe71b3ea2d2f3a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:14.707 INFO 2458 --- [prometheus-minute-schedule-1,a6500653b936a2e4,1a9b1006773d1be6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:14.714 INFO 2458 --- [prometheus-minute-schedule-1,a6500653b936a2e4,1a9b1006773d1be6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:15.492 INFO 2458 --- [ocp-schedules-2,427c957fcf88e7e0,438e2f60f5904dc2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:15.497 INFO 2458 --- [ocp-schedules-2,427c957fcf88e7e0,438e2f60f5904dc2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:15.574 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:15.574 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:16.089 INFO 2458 --- [ocp-schedules-17,0c2921585105650e,910dd58e07eca999] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:32:16.089+08:00, clusters=[1, 2] 2025-10-27 11:32:16.094 INFO 2458 --- [ocp-schedules-17,0c2921585105650e,910dd58e07eca999] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:32:16.095 INFO 2458 --- [ocp-schedules-17,0c2921585105650e,910dd58e07eca999] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:32:16.636 INFO 2458 --- [ocp-async-8,c9f611c9e3c16fe4,93a184d02d132d44] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:16.636 WARN 2458 --- [ocp-async-5,55e77252621446bc,cf89fe745734f73e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:17.005 INFO 2458 --- [ocp-schedules-1,8f8e6751b5cedcb2,3e802ac15846ea26] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:17.444 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=1000, affectRows=1000 2025-10-27 11:32:17.448 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=999, validCount=995 2025-10-27 11:32:17.487 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535860, nodeCount=817, affectRows=817 2025-10-27 11:32:17.503 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=1000, affectRows=1000 2025-10-27 11:32:17.507 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=996, validCount=992 2025-10-27 11:32:17.519 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=514, affectRows=514 2025-10-27 11:32:17.521 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=511, validCount=508 2025-10-27 11:32:17.524 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=565, affectRows=565 2025-10-27 11:32:17.527 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=562, validCount=561 2025-10-27 11:32:17.707 INFO 2458 --- [prometheus-minute-schedule-1,2cfb25de291f30b0,ee08cd0aa1dd58eb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:17.714 INFO 2458 --- [prometheus-minute-schedule-1,2cfb25de291f30b0,ee08cd0aa1dd58eb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:17.958 INFO 2458 --- [ocp-schedules-27,6448c1ec7839159f,2ce227585e1eb11e] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:32:17.958 INFO 2458 --- [ocp-schedules-27,6448c1ec7839159f,2ce227585e1eb11e] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:32:18.010 INFO 2458 --- [Collect all tenants major compactions2,6448c1ec7839159f,9c9a72207b050c9d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:32:18.011 INFO 2458 --- [Collect all tenants major compactions3,6448c1ec7839159f,ec35a4871a8794cb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:32:18.011 INFO 2458 --- [Collect all tenants major compactions4,6448c1ec7839159f,62b294dc125c5325] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:32:18.012 INFO 2458 --- [Collect all tenants major compactions5,6448c1ec7839159f,870f1d326dfba826] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:32:18.012 INFO 2458 --- [Collect all tenants major compactions1,6448c1ec7839159f,248dde33439641b9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:32:18.013 INFO 2458 --- [Collect all tenants major compactions6,6448c1ec7839159f,947c79e6dcbdc15a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:32:18.048 INFO 2458 --- [Collect all tenants major compactions6,6448c1ec7839159f,947c79e6dcbdc15a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.048 INFO 2458 --- [Collect all tenants major compactions5,6448c1ec7839159f,870f1d326dfba826] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.050 INFO 2458 --- [Collect all tenants major compactions5,6448c1ec7839159f,870f1d326dfba826] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:32:18.049 INFO 2458 --- [Collect all tenants major compactions6,6448c1ec7839159f,947c79e6dcbdc15a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:32:18.054 INFO 2458 --- [Collect all tenants major compactions5,6448c1ec7839159f,870f1d326dfba826] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:32:18.054 INFO 2458 --- [Collect all tenants major compactions6,6448c1ec7839159f,947c79e6dcbdc15a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:32:18.065 INFO 2458 --- [Collect all tenants major compactions2,6448c1ec7839159f,9c9a72207b050c9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.065 INFO 2458 --- [Collect all tenants major compactions4,6448c1ec7839159f,62b294dc125c5325] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.066 INFO 2458 --- [Collect all tenants major compactions3,6448c1ec7839159f,ec35a4871a8794cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.066 INFO 2458 --- [Collect all tenants major compactions2,6448c1ec7839159f,9c9a72207b050c9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:32:18.067 INFO 2458 --- [Collect all tenants major compactions4,6448c1ec7839159f,62b294dc125c5325] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:32:18.067 INFO 2458 --- [Collect all tenants major compactions1,6448c1ec7839159f,248dde33439641b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:18.068 INFO 2458 --- [Collect all tenants major compactions3,6448c1ec7839159f,ec35a4871a8794cb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:32:18.070 INFO 2458 --- [Collect all tenants major compactions1,6448c1ec7839159f,248dde33439641b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:32:18.071 INFO 2458 --- [Collect all tenants major compactions2,6448c1ec7839159f,9c9a72207b050c9d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:32:18.071 INFO 2458 --- [Collect all tenants major compactions4,6448c1ec7839159f,62b294dc125c5325] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:32:18.071 INFO 2458 --- [Collect all tenants major compactions3,6448c1ec7839159f,ec35a4871a8794cb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:32:18.073 INFO 2458 --- [Collect all tenants major compactions1,6448c1ec7839159f,248dde33439641b9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:32:18.074 INFO 2458 --- [ocp-schedules-27,6448c1ec7839159f,2ce227585e1eb11e] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:32:18.539 INFO 2458 --- [ocp-async-10,be8a25893ea4cac0,916c119f21d05cdc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:18.574 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:18.576 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:18.636 INFO 2458 --- [ocp-async-6,3674c744c637da55,f4c0f04696b1acf5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:18.636 WARN 2458 --- [ocp-async-9,fb9a16edeb5ad84b,a46f7136c2c1c116] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:20.588 INFO 2458 --- [ocp-schedules-20,662506538a010bdb,130ca0e0fc33473a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:32:20.637 INFO 2458 --- [ocp-async-6,2f2111c4ccabee02,afc7c268a0aba765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:20.637 WARN 2458 --- [ocp-async-1,07297f994e816a27,25239e0d6a3e9eb4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:20.707 INFO 2458 --- [prometheus-minute-schedule-1,03c8fac1dd1bba30,efc1235cc71d97d3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:20.714 INFO 2458 --- [prometheus-minute-schedule-1,03c8fac1dd1bba30,efc1235cc71d97d3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:21.410 INFO 2458 --- [ocp-schedules-17,c91ab38b3c676006,16cc3bc1736faf86] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:32:21.410 INFO 2458 --- [ocp-schedules-27,9e56845a3cc9e9ef,d3476c4938fb7198] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:32:21.410 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:32:21.411 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:32:21.411 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:32:21.411 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:32:21.411 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:32:21.425 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:21.425 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:21.425 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:21.425 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:21.426 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:21.426 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:21.428 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:21.430 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:21.431 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:21.431 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:21.431 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:21.432 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:21.432 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:21.432 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:21.432 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:21.433 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:21.433 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:21.435 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:21.437 INFO 2458 --- [ocp-schedules-8,4532eec7597d18ca,dec5e2602e488e39] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:21.437 INFO 2458 --- [host-check-clockdiff1,4532eec7597d18ca,27164a8746a5faa9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-clockdiff2,4532eec7597d18ca,9c01810343c94deb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-clockdiff3,4532eec7597d18ca,84f7d5e778c7bb7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-clockdiff4,4532eec7597d18ca,898ac81ceef9a1df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-clockdiff5,4532eec7597d18ca,6a232ac873f6081f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:21.438 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:21.442 INFO 2458 --- [ocp-schedules-6,01216357b68bc9fa,c444a2dff6afc30a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:32:21.458 INFO 2458 --- [host-check-clockdiff4,4532eec7597d18ca,898ac81ceef9a1df] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535941 -1 -1 , error= 2025-10-27 11:32:21.470 INFO 2458 --- [host-check-clockdiff5,4532eec7597d18ca,6a232ac873f6081f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535941 0 0 , error= 2025-10-27 11:32:21.470 INFO 2458 --- [host-check-clockdiff1,4532eec7597d18ca,27164a8746a5faa9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535941 0 0 , error= 2025-10-27 11:32:21.470 INFO 2458 --- [host-check-clockdiff3,4532eec7597d18ca,84f7d5e778c7bb7b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535941 -1 -1 , error= 2025-10-27 11:32:21.484 INFO 2458 --- [host-check-clockdiff2,4532eec7597d18ca,9c01810343c94deb] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535941 -1 -1 , error= 2025-10-27 11:32:21.512 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:32:21.512 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:32:21.512 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:32:21.514 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:32:21.514419845+08:00 2025-10-27 11:32:21.515 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:32:21.515+08:00 2025-10-27 11:32:21.516 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:32:21.516 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:32:21.516 INFO 2458 --- [host-check-agent-status3,c91ab38b3c676006,f7076415671d9876] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:32:21.517 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:32:21.517 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:32:21.517 INFO 2458 --- [host-check-agent-status1,c91ab38b3c676006,efa4acfbcb156007] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:32:21.517 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:32:21.518 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:32:21.518137580+08:00 2025-10-27 11:32:21.518 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:32:21.518 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:32:21.518+08:00 2025-10-27 11:32:21.520 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:32:21.519246433+08:00 2025-10-27 11:32:21.521 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:32:21.520+08:00 2025-10-27 11:32:21.521 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:32:21.524 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:32:21.524 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:32:21.524 INFO 2458 --- [host-check-agent-status5,c91ab38b3c676006,b7670a623ddd3cd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:32:21.524 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:32:21.515+08:00 and status AVAILABLE 2025-10-27 11:32:21.525 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:32:21.526 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:32:21.518+08:00 and status AVAILABLE 2025-10-27 11:32:21.528 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:32:21.528 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:32:21.528 INFO 2458 --- [host-check-agent-status2,c91ab38b3c676006,a0f8c0076c7cf5b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:32:21.528 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:32:21.530 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:32:21.520+08:00 and status AVAILABLE 2025-10-27 11:32:21.530 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:32:21.531 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:32:21.531 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-agent-status4,c91ab38b3c676006,c603f36992cffc5d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:32:21.531822465+08:00 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:32:21.531627130+08:00 2025-10-27 11:32:21.532 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:32:21.532+08:00 2025-10-27 11:32:21.533 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:32:21.532+08:00 2025-10-27 11:32:21.533 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:32:21.535 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:32:21.540 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:32:21.532+08:00 and status AVAILABLE 2025-10-27 11:32:21.542 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:32:21.532+08:00 and status AVAILABLE 2025-10-27 11:32:21.545 INFO 2458 --- [ocp-schedules-11,d45aa47dcd02ad3c,a67204ffb4833e54] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:32:21.546 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:32:21.548 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:32:21.549 INFO 2458 --- [ocp-schedules-25,9052873b8abb92cb,2f8477c40d2c10ea] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:32:21.549 INFO 2458 --- [ocp-schedules-11,d45aa47dcd02ad3c,a67204ffb4833e54] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:21.553 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:21.553 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:21.554 INFO 2458 --- [ocp-schedules-12,6a674ddd8b7d2bcb,6432c4b2db7fde23] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:32:21.555 INFO 2458 --- [ocp-schedules-25,9052873b8abb92cb,2f8477c40d2c10ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:21.555 INFO 2458 --- [ocp-schedules-25,9052873b8abb92cb,2f8477c40d2c10ea] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:32:21.556 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:21.557 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:21.557 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:21.558 INFO 2458 --- [ocp-schedules-12,6a674ddd8b7d2bcb,6432c4b2db7fde23] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:21.559 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:21.570 INFO 2458 --- [ocp-schedules-13,b0667c4b0f924603,39ca056bd9541ac6] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:32:21.573 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:21.573 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:21.576 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:21.576 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:21.577 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:21.577 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:21.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:21.602 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:32:21.618 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.618 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:21.618 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:21.624 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:32:21.625 INFO 2458 --- [ocp-schedules-11,d45aa47dcd02ad3c,a67204ffb4833e54] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:32:21.625 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:21.626 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:21.631 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.631 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:21.634 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:21.635 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:21.635 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:32:21.645 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:32:21.650 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:32:21.655 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:32:21.683 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:32:21.683 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.685 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:32:21.689 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:32:21.690 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:32:21.691 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:32:21.705 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.705 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:21.716 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:32:21.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:21.719739+08:00, lastProcessTime=2025-10-27T11:32:11.722027+08:00 2025-10-27 11:32:21.722 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:21.724 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:21.725 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:21.725 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:21.739 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:32:21.742 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:32:21.744 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:32:21.745 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:32:21.746 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:32:21.758 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:21.759 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.759 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:21.761 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:32:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:21.770 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:32:21.773 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:32:21.775 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:32:21.775 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:21.777 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:32:21.783 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:21.783 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.785 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:32:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:21.803 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:21.806 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:32:21.808 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:21.813 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:21.816 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:21.816 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:21.820 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:32:21.830 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:32:21.833 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:32:21.834 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:32:21.838 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:32:21.843 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:21.844 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.845 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:21.846 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:32:21.854 INFO 2458 --- [host-check-host-status4,9e56845a3cc9e9ef,7382fb806c76cccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:32:21.857 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:32:21.864 INFO 2458 --- [host-check-host-status5,9e56845a3cc9e9ef,e97acf008f483453] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:32:21.867 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:21.868 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:21.869 INFO 2458 --- [host-check-host-status2,9e56845a3cc9e9ef,2320ea5261f9a12f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:32:21.870 INFO 2458 --- [host-check-host-status3,9e56845a3cc9e9ef,a0817756658630c8] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:32:21.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:21.871 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:21.872 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.874 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.879 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:32:21.880 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:21.894 INFO 2458 --- [ocp-schedules-29,966bd7fa9575c705,482f87947abc9b2a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:32:21.898 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:32:21.924 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:21.925 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:21.929 INFO 2458 --- [host-check-host-status1,9e56845a3cc9e9ef,8406d46148d3cd88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:32:21.935 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:32:21.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:21.963 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:21.970 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:21.719739+08:00 2025-10-27 11:32:21.979 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:32:21.980 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.980 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:32:21.982 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:21.983 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:32:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:21.998 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:32:22.023 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:22.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:22.024 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:32:22.035 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:22.035 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:32:22.048 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:32:22.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:22.070 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:22.101 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535860, nodeCount=1000, affectRows=1000 2025-10-27 11:32:22.105 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535860, nodeCount=1000, validCount=987 2025-10-27 11:32:22.108 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:22.151 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:22.151 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:22.155 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:32:22.164 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:32:22.175 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:22.178 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:22.227 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:32:22.298 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:32:22.298 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:22.299 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:32:22.301 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:22.302 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:32:22.335 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:32:22.377 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:22.378 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:22.379 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:32:22.418 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:32:22.418 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:32:22.418 INFO 2458 --- [ocp-schedules-12,6a674ddd8b7d2bcb,6432c4b2db7fde23] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:32:22.463 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:22.463 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=3 2025-10-27 11:32:22.463 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=4 2025-10-27 11:32:22.464 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=2 2025-10-27 11:32:22.464 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=1 2025-10-27 11:32:22.464 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Refreshing OCP agent configuration: hostId=5 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_retry_remove_parts, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fail_check_localserver, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mem_alloc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_hang_due_to_trx_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_parse_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_refresh_schema_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_alloc_oops, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_invalid_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_assemble_rpc_lock, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_global_trans_version_delay, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_local_ip_invalid, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_common_unexpected_internal_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_unsync, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_major_freeze_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_distributed_strong_read, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_reply_memstore_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_write_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType rootservice_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_error_lib_direct_alloc, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_fetch_rs_url_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_partition_overlimit, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_rootservice_destroy_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_max_open_files, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_worker_thread_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_mysql_request_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_trx_context_not_release, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_checksum_mismatch, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_replica_minority, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unknown_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_online_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_no_valid_parti, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_meta_inconsistent, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_network_unreachable, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_tx_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_too_many_sstables, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_right_to_die, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_cant_daily_merge, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_voting_success, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_virtual_group_elect_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_thread_hang, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_microblock_read0, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_merge_error_4184, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_dump_queue_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType election_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_unit_migrate_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_commit_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_easy_hold_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_row_count_notsame, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_restart_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_io_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_server_disk_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_merge_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType observer_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_disk_almost_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_timer_cost_much, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clog_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_decent_voting, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_get_events_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rs_checksum_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_blocklist, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_clock_diff_time_too_large, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_service_start_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_system_disk_full, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType refresh_location_cache_failed, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_crash_error, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.553 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_ps_checksum_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_rpc_fly_slow, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_err_defensive_check, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_log_renew_location_timeout, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.554 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType ob_server_stop_fail, alarm targets do not contain hostIp 10.40.65.151 2025-10-27 11:32:22.568 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562} 2025-10-27 11:32:22.572 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 4 agent sql collect property 2025-10-27 11:32:22.572 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:22.581 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:22.585 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStan 2025-10-27 11:32:22.586 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:22.592 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:22.592 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:32:22.592 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:22.593 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:32:22.593 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:32:22.593 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.66.107 2025-10-27 11:32:22.601 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 5 agent sql collect property 2025-10-27 11:32:22.601 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:22.604 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:22.604 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:32:22.604 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:22.608 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:22.616 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:22.616 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:32:22.616 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:22.629 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:32:22.629 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:32:22.629 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.168 2025-10-27 11:32:22.635 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 2 agent sql collect property 2025-10-27 11:32:22.637 INFO 2458 --- [ocp-async-7,9237d15906d213c3,c9abf9b101bb28ac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:22.637 WARN 2458 --- [ocp-async-8,8b95d4a7cea0a1a2,2975a041af0ca107] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:22.633 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_query_slow, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:32:22.637 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType odp_sql_execute_failed, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:32:22.638 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.alarm.service.OcpAlarmRuleService : alarmType obproxy_log_alarm, alarm targets do not contain hostIp 10.40.65.146 2025-10-27 11:32:22.642 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 1 agent sql collect property 2025-10-27 11:32:22.648 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:22.649 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 2 to set sql collect property inactive 2025-10-27 11:32:22.655 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:22.655 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:22.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 1 to set sql collect property inactive 2025-10-27 11:32:22.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:22.663 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:22.672 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] o.p.s.i.SqlCollectHostPropertiesProvider : start to set hostId 3 agent sql collect property 2025-10-27 11:32:22.674 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:22.689 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (logtailer.log.filter.rules.json.content=[{"alarmTemplateId":0,"alarmType":"message_hardware_error","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["Hardware Error"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"message_log_alarm","durationSeconds":0,"evaluationIntervalSeconds":0,"excludeKeywords":[],"excludeKeywordsRelation":"Or","expression":{"metric":"os_message_log"},"expressionType":"Log","includeKeywords":["kernel: nvme.*timeout"],"includeKeywordsRelation":"Or","level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_compress_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_compress_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300},{"alarmTemplateId":0,"alarmType":"odp_instance_expanse_failed","durationSeconds":0,"evaluationIntervalSeconds":60,"excludeKeywords":[],"expression":{"metric":"instance_expanse_failed_count","op":">","threshold":0.00},"expressionType":"LogStat","includeKeywords":[],"level":2,"logLevels":[],"resolvedTimeoutSeconds":300}]), (monagent.pipeline.ob.trace.status=inactive), (monagent.pipeline.ob.log.status=inactive)]), params:null 2025-10-27 11:32:22.693 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] o.p.s.i.SqlCollectHostPropertiesProvider : hostsId 3 to set sql collect property inactive 2025-10-27 11:32:22.697 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:22.702 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:22.721 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 4, do nothing 2025-10-27 11:32:22.758 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510188), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.66.107), (ob.run.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=2), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.66.107:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase2), (ob.logtailer.enabled=true), (ob.l 2025-10-27 11:32:22.774 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone1), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.146), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.146:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:32:22.778 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 5, do nothing 2025-10-27 11:32:22.780 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone3), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.65.168), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.65.168:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtail 2025-10-27 11:32:22.794 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 1, do nothing 2025-10-27 11:32:22.799 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 2, do nothing 2025-10-27 11:32:22.808 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/module/config/update, request body:UpdateAgentConfigRequest(agentHome=null, configs=[(ocp.agent.snapshot.maxsize=1GB), (ocp.agent.snapshot.maxage.days=90), (agent.http.basic.auth.metricAuthEnabled=true), (monagent.second.metric.cache.update.interval=15s), (monagent.log.level=info), (ocp.agent.collect.snapshot.status=active), (mgragent.log.level=info), (monagent.ob.zone.name=zone2), (ob.logcleaner.ob_log.rule1.file.regex=([a-z]+.)?[a-z]+.log.wf.[0-9]+), (monagent.ob.err.code.prefix=errcode=-), (monagent.ob.cgroup.status=active), (monagent.pipeline.ob.status=active), (ob.run.user=admin), (monagent.monitordb.host=10.40.65.146), (ob.logcleaner.ob_log.rule0.retention.days=8), (ob.sync.status=VALID), (monagent.ob.cluster.id=1760510187), (ob.logcleaner.ob_log.disk.threshold=80), (ob.logcleaner.ob_log.rule2.file.regex=[a-z]+.log.[0-9]+.zst), (monagent.monitordb.mysql.address=tcp(10.40.65.146:2883)), (ob.logcleaner.ob_log.rule1.keep.percentage=80), (ob.logcleaner.core_log.rule0.keep.percentage=60), (monagent.monitordb.password=xxx (monagent.monitordb.database=monitor_database), (monagent.ob.rpc.port=2882), (ob.logcleaner.core_log.disk.threshold=80), (monagent.host.ip=10.40.67.22), (ob.run.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule2.keep.percentage=80), (ob.install.path=/home/admin/myoceanbase/oceanbase), (ob.logcleaner.ob_log.rule1.retention.days=30), (ob.logcleaner.core_log.rule0.file.regex=core.[0-9]+), (ob.data.path=/opt/oceanbase/data), (monagent.cluster.id=1), (monagent.ob.sql.port=2881), (monagent.monitordb.port=2883), (monagent.ob.monitor.password=xxx (ob.log.path=/opt/oceanbase/log), (ob.logcleaner.enabled=true), (ob.logcleaner.core_log.rule0.retention.days=8), (monagent.pipeline.awr.collect.disabled=true), (monagent.ob.monitor.mysql.address=tcp(10.40.67.22:2881)), (ob.logcleaner.ob_log.rule0.file.regex=[a-z]+.[a-z]+.log.[0-9]+), (ob.logcleaner.ob_log.rule0.keep.percentage=60), (monagent.ob.cluster.name=myoceanbase), (ob.logtailer. 2025-10-27 11:32:22.825 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449542825000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:32:22.825Z, endTime=2025-10-27T03:30:22.562Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:32:22.828 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449542828000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:32:22.828Z, endTime=2025-10-27T03:30:22.562Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:32:22.828 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.c.a.AgentConfigChangeServiceImpl : no change for host 3, do nothing 2025-10-27 11:32:22.830 INFO 2458 --- [ocp-schedules-15,8fcdd337a68cf27b,02596aab24bf9e03] c.o.o.s.common.scheduled.HostSchedules : Refresh all agent config finished 2025-10-27 11:32:22.837 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:32:22.837 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:32:22.871 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 264 2025-10-27 11:32:22.872 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:32:22.873 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:32:22.874 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:32:22.882 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:32:22.882 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:32:22.895 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:32:22.895 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:32:22.903 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:32:22.903 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:32:22.908 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:32:22.908 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:32:22.911 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:32:22.911 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:32:22.828, end=2025-10-27T11:30:22.562}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:32:22.925 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:32:22.925 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535822562000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:32:22.930 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:32:23.275 INFO 2458 --- [plan_change_detector2,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:32:22.825, end=2025-10-27T11:30:22.562}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:32:23.275 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:23.277 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:23.278 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:32:23.278 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:23.280 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:23.282 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:23.282 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:32:23.282 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:23.284 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:32:23.285 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:28:21.63, end=2025-10-27T11:30:22.562},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:32:23.285 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:32:23.285 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:32:23.287 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:32:23.707 INFO 2458 --- [prometheus-minute-schedule-1,468b80183f3cbd1a,27967791058ec0b6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:23.716 INFO 2458 --- [prometheus-minute-schedule-1,468b80183f3cbd1a,27967791058ec0b6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:23.897 INFO 2458 --- [ocp-schedules-30,ddb6010d5f912f57,82004130a257ab2d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:23.947 WARN 2458 --- [scanner-112206113-1,ddb6010d5f912f57,bc471fec04da1d2f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:24.577 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:24.578 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:24.637 INFO 2458 --- [ocp-async-7,5bd96f7857fa5cff,9ab56573cc8147c5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:24.637 WARN 2458 --- [ocp-async-4,01e3ebbe5c990492,722e2748ca0eee15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:25.504 INFO 2458 --- [ocp-schedules-9,be31397ac1a0bb99,aedc8b79fe1a5f81] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:25.508 INFO 2458 --- [ocp-schedules-9,be31397ac1a0bb99,aedc8b79fe1a5f81] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:26.637 WARN 2458 --- [ocp-async-3,3c549ae358e9f1da,c7eb0fa6cf08c1d9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:26.637 INFO 2458 --- [ocp-async-1,d0b23c35f97e618e,dabdc09bed055dd8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:26.707 INFO 2458 --- [prometheus-minute-schedule-1,94243dc4a4908c39,0a14baa64b2bd67c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:26.714 INFO 2458 --- [prometheus-minute-schedule-1,94243dc4a4908c39,0a14baa64b2bd67c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:27.011 INFO 2458 --- [ocp-schedules-28,81e5863845a995e0,a36dacf28cf8e0b0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:27.578 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:27.578 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:27.664 INFO 2458 --- [ocp-schedules-6,16bb41e0d002d1e8,af44d9443340453d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:32:27.672 INFO 2458 --- [ocp-schedules-6,16bb41e0d002d1e8,af44d9443340453d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:27.672 INFO 2458 --- [ocp-schedules-6,16bb41e0d002d1e8,af44d9443340453d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:28.539 INFO 2458 --- [ocp-async-1,03dd0865f72f4eb3,6300151f096beabd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:28.637 WARN 2458 --- [ocp-async-6,c75bb271074b9692,e38fa92aecd8f666] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:28.637 INFO 2458 --- [ocp-async-8,8a4ae8c179ea3846,8637567594203ccb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:29.052 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:32:29.052 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:32:29.054 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:32:29.054 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:32:29.055 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:32:29.055 INFO 2458 --- [ocp-schedules-17,b85592e989088e3e,f700807e26d8fc84] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:32:29.496 INFO 2458 --- [ocp-schedules-26,5c63e73cdf8dcbd8,39213322b101eecb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:32:29.502 INFO 2458 --- [ocp-schedules-26,5c63e73cdf8dcbd8,39213322b101eecb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:32:29.502 INFO 2458 --- [ocp-schedules-26,5c63e73cdf8dcbd8,39213322b101eecb] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:32:29.707 INFO 2458 --- [prometheus-minute-schedule-1,a238b943b4e90c4d,01385f9a7ff5ad96] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:29.715 INFO 2458 --- [prometheus-minute-schedule-1,a238b943b4e90c4d,01385f9a7ff5ad96] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:30.579 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:30.580 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:30.637 WARN 2458 --- [ocp-async-9,2df002c06fb47e8c,e1480bd0041ed1ff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:30.637 INFO 2458 --- [ocp-async-4,230b3c78d6140d42,28453ae972c2e7a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:31.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:31.717595+08:00, lastProcessTime=2025-10-27T11:32:21.719739+08:00 2025-10-27 11:32:31.928 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:31.717595+08:00 2025-10-27 11:32:32.637 INFO 2458 --- [ocp-async-3,7560857725eaf182,23ca6d1c161ad279] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:32.638 WARN 2458 --- [ocp-async-2,13466c727ce3ed89,3c16672221619583] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:32.707 INFO 2458 --- [prometheus-minute-schedule-1,e3fdfb97373fee3f,4d3754561284c4e3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:32.714 INFO 2458 --- [prometheus-minute-schedule-1,e3fdfb97373fee3f,4d3754561284c4e3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:33.580 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:33.581 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:33.967 INFO 2458 --- [ocp-schedules-4,0515a4d85adfbb2c,0c86cbd9c2005adf] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:32:33.967 INFO 2458 --- [ocp-schedules-4,0515a4d85adfbb2c,0c86cbd9c2005adf] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:32:34.086 INFO 2458 --- [ocp-schedules-2,9cb1812946d3745a,19d0a2250304a6b7] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:34.101 WARN 2458 --- [scanner-112206362-1,9cb1812946d3745a,bc7471d20e355079] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:34.191 INFO 2458 --- [alarm-detect-14,9cb1812946d3745a,2e343f976a642244] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535920 ~ 1761535944 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535935, value=119.329)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535935, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535935=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:32:34.193 INFO 2458 --- [alarm-detect-14,9cb1812946d3745a,2e343f976a642244] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:32:34.638 WARN 2458 --- [ocp-async-6,ae289a1f9b522adb,e7d012b94135433a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:34.638 INFO 2458 --- [ocp-async-5,9d07aab7d28969a0,aaaa2860d906c749] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:35.515 INFO 2458 --- [ocp-schedules-15,8e08e257c3490a4a,efa6ba99ba8d37f4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:35.519 INFO 2458 --- [ocp-schedules-15,8e08e257c3490a4a,efa6ba99ba8d37f4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:35.707 INFO 2458 --- [prometheus-minute-schedule-1,9af9f8c68c3300fe,1afd82deaf0f3801] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:35.714 INFO 2458 --- [prometheus-minute-schedule-1,9af9f8c68c3300fe,1afd82deaf0f3801] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:36.581 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:36.582 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:36.638 WARN 2458 --- [ocp-async-9,c5d377f7fa4e2c4d,9919837a2cbd21a9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:36.638 INFO 2458 --- [ocp-async-8,770af87831b6e933,6195aeb64e9ae733] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:37.018 INFO 2458 --- [ocp-schedules-5,99b8844d76996dd3,36174596a2634451] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:37.708 INFO 2458 --- [prometheus-cache-monitor1,408a4602c3461a67,87c260cdf58b6eb7] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20331,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5078,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:32:37.708 INFO 2458 --- [prometheus-cache-monitor1,408a4602c3461a67,87c260cdf58b6eb7] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:32:38.540 INFO 2458 --- [ocp-async-8,72abfda52e3a32be,1b2aa893800975f5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:38.638 WARN 2458 --- [ocp-async-2,4bc6a1692773bf22,2eb2ca2136aada49] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:38.638 INFO 2458 --- [ocp-async-1,7aa62e50f230f643,145923d9cf3fb4e9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:38.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:32:38.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:32:38.707 INFO 2458 --- [prometheus-minute-schedule-1,40b50374ffad8f15,e28e8b94a9afb302] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:38.711 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:32:38.711 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:32:38.711 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:32:38.711 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:32:38.714 INFO 2458 --- [prometheus-minute-schedule-1,40b50374ffad8f15,e28e8b94a9afb302] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:38.715 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:32:38.715 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:32:38.715 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:32:38.715 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:32:38.743 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:32:38.749 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:32:38.749 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:32:38.749 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:32:38.749 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:32:38.749 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:32:38.755 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:32:38.825 INFO 2458 --- [ocp-schedules-28,86820e8b2963e874,5643345480ab09bc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:32:38.833 INFO 2458 --- [ocp-schedules-28,86820e8b2963e874,5643345480ab09bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:38.834 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:32:38.834 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:32:38.834 INFO 2458 --- [ocp-schedules-28,86820e8b2963e874,5643345480ab09bc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:32:39.152 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:32:39.530 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.814505371S 2025-10-27 11:32:39.530 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:32:39.584 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:39.586 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:39.619 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:32:39.619 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:32:39.661 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:32:39.707 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:32:39.715 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:32:39.841 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] o.s.boot.SpringApplication : Started application in 1.13 seconds (JVM running for 258428.268) 2025-10-27 11:32:39.841 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.130667984S 2025-10-27 11:32:39.845 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:39.845 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:32:39.845 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:32:39.845 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:32:39.845 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:32:39.845 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:32:39.857 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:39.859 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:39.859 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:39.860 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:39.860 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:39.863 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:39.863 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:39.864 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:39.864 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:39.864 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:39.915 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:32:39.915 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:32:39.917 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:32:39.917 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:32:39.926 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:32:39.927 INFO 2458 --- [ocp-schedules-18,684e256103dd79fb,8a1e87719dcb824d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:32:39.976 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:32:39.976 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:32:40.010 INFO 2458 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:32:40.011 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:32:40.044 INFO 2458 --- [Check obproxy1,00559b9429aed7fe,efc6b808c6e69d7b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:32:40.048 INFO 2458 --- [Check obproxy1,00559b9429aed7fe,efc6b808c6e69d7b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:32:40.050 INFO 2458 --- [Collect obproxy1,619fdd1eb2919029,46fcb8769f639ff8] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:32:40.051 INFO 2458 --- [Collect obproxy1,619fdd1eb2919029,46fcb8769f639ff8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:32:40.078 INFO 2458 --- [Check obproxy1,00559b9429aed7fe,efc6b808c6e69d7b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:32:40.081 INFO 2458 --- [Check obproxy1,00559b9429aed7fe,efc6b808c6e69d7b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:32:40.081 INFO 2458 --- [Check obproxy1,00559b9429aed7fe,efc6b808c6e69d7b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:32:40.088 INFO 2458 --- [Collect obproxy1,619fdd1eb2919029,46fcb8769f639ff8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:32:40.091 INFO 2458 --- [Collect obproxy1,619fdd1eb2919029,46fcb8769f639ff8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:32:40.091 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:40.092 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:32:40.094 INFO 2458 --- [Collect obproxy1,619fdd1eb2919029,46fcb8769f639ff8] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:32:40.116 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:40.117 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:32:40.182 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:40.183 INFO 2458 --- [ocp-schedules-17,72e3050a5297f5f5,dd6b3b2ebe50cff9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:32:40.222 INFO 2458 --- [ocp-schedules-1,c57a152cabdb94d0,566b07f92372c87b] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:32:40.638 INFO 2458 --- [ocp-async-6,9a45b562984b74bf,6b62e56b962bed1b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:40.638 WARN 2458 --- [ocp-async-5,ebd89c8ae852599e,5189c64e8af6fdc2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:41.281 INFO 2458 --- [ocp-schedules-10,0a49a7957ca769b0,f8e89202fec10995] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:32:41.707 INFO 2458 --- [prometheus-minute-schedule-1,f267af3b218dfbbb,ed1895e50454d3a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:41.715 INFO 2458 --- [prometheus-minute-schedule-1,f267af3b218dfbbb,ed1895e50454d3a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:41.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:41.717289+08:00, lastProcessTime=2025-10-27T11:32:31.717595+08:00 2025-10-27 11:32:41.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:41.717289+08:00 2025-10-27 11:32:42.586 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:42.587 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:42.638 INFO 2458 --- [ocp-async-9,0a0efba13849e819,e27b2c1e1e1fbb67] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:42.638 WARN 2458 --- [ocp-async-7,d6b6070556344d90,8957471326b40062] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:44.232 INFO 2458 --- [ocp-schedules-26,3e9a422d28f0a57e,c3d3bf82939c195a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:44.272 INFO 2458 --- [ocp-schedules-10,083c17dea5f1ad1d,de100ce9e0562362] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:32:44.280 WARN 2458 --- [scanner-112206548-1,3e9a422d28f0a57e,20b8260d260c2cda] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:44.433 INFO 2458 --- [alarm-detect-7,3e9a422d28f0a57e,3374ca412d8bdee3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535935 ~ 1761535954 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535950, value=86.393)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535950, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535950=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:32:44.638 INFO 2458 --- [ocp-async-4,d918a5c0c93b7560,b624fb51897a71a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:44.638 WARN 2458 --- [ocp-async-3,4b426d6add75b13d,1c9ee9b23a56c538] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:44.707 INFO 2458 --- [prometheus-minute-schedule-1,8cd2d4b235dc995e,b1912525b6f0f4b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:44.713 INFO 2458 --- [prometheus-minute-schedule-1,8cd2d4b235dc995e,b1912525b6f0f4b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:45.102 INFO 2458 --- [prometheus-second-schedule-1,2b4f7af7e32bcc5b,cf1cc59f31673431] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:32:45.527 INFO 2458 --- [ocp-schedules-16,b721cc86bd115a9d,5640a088a92acde9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:45.531 INFO 2458 --- [ocp-schedules-16,b721cc86bd115a9d,5640a088a92acde9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:45.587 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:45.588 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:46.638 WARN 2458 --- [ocp-async-1,2420016d81de984e,84247acd9978de26] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:46.638 INFO 2458 --- [ocp-async-1,c78736bd079eb6e8,e33e3a2edfc6d1c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:47.024 INFO 2458 --- [ocp-schedules-25,25c094a5870285a3,28e77dea4bcab2ec] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:47.674 INFO 2458 --- [ocp-schedules-15,6d1f8e94c6ebf413,93b115ce2eb5cfcd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:32:47.684 INFO 2458 --- [ocp-schedules-15,6d1f8e94c6ebf413,93b115ce2eb5cfcd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:47.684 INFO 2458 --- [ocp-schedules-15,6d1f8e94c6ebf413,93b115ce2eb5cfcd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:32:47.707 INFO 2458 --- [prometheus-minute-schedule-1,8fb3e1efe63d8bd0,928d6313ac4d5e24] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:47.714 INFO 2458 --- [prometheus-minute-schedule-1,8fb3e1efe63d8bd0,928d6313ac4d5e24] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:47.971 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:32:47.993 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:47.997 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:48.047 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:48.050 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:48.095 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:48.098 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:48.145 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:48.148 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:48.199 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:48.202 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:48.251 INFO 2458 --- [ocp-schedules-2,18ba560d260da35b,a7915bc6ffc20bc1] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:32:48.540 INFO 2458 --- [ocp-async-1,aa9a04eaeba9ba02,4551cbc246eeeb18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:48.588 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:48.589 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:48.639 WARN 2458 --- [ocp-async-7,b3ba62a0b6c84c06,f59e993f1582408e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:48.639 INFO 2458 --- [ocp-async-8,47b500f17a5e5090,55e83e346a002e18] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:49.801 INFO 2458 --- [ocp-schedules-3,6fa5e03bb90fd3be,f491ea79eaeb41f9] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:32:49.801+08:00, clusters=[1, 2] 2025-10-27 11:32:49.818 INFO 2458 --- [ocp-schedules-3,6fa5e03bb90fd3be,f491ea79eaeb41f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:32:49.820 INFO 2458 --- [ocp-schedules-3,6fa5e03bb90fd3be,f491ea79eaeb41f9] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:32:50.639 INFO 2458 --- [ocp-async-3,8d79347a8e818f47,ac8b918c30cd153e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:50.639 WARN 2458 --- [ocp-async-2,3c058374c804531e,ca903838031cd2c9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:50.707 INFO 2458 --- [prometheus-minute-schedule-1,f356062bd699a7fa,034c4295a00d75a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:50.714 INFO 2458 --- [prometheus-minute-schedule-1,f356062bd699a7fa,034c4295a00d75a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:51.408 INFO 2458 --- [ocp-schedules-12,a20598a15f8c31bb,a442739b32331b33] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:51.408 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:32:51.408 INFO 2458 --- [ocp-schedules-4,bc11b5c9171d9e48,cdb02c3cb6d23c00] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:51.408 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:32:51.409 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:32:51.410 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:32:51.410 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:32:51.423 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:51.423 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:51.424 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:51.425 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:51.426 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:51.426 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:51.427 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:51.428 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:51.428 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:51.428 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:51.429 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:51.429 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:51.429 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:51.432 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:51.432 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:51.432 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:51.432 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:51.435 INFO 2458 --- [ocp-schedules-5,ecfc974a08688f94,674d11f77a245421] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:32:51.436 INFO 2458 --- [host-check-clockdiff1,ecfc974a08688f94,7aeb68cfef1175a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:32:51.436 INFO 2458 --- [host-check-clockdiff2,ecfc974a08688f94,41028fc8d87fa643] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:32:51.436 INFO 2458 --- [host-check-clockdiff3,ecfc974a08688f94,8753058778b8e232] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:32:51.436 INFO 2458 --- [host-check-clockdiff4,ecfc974a08688f94,3ed9a28a85001b31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:32:51.436 INFO 2458 --- [host-check-clockdiff5,ecfc974a08688f94,71d901e73bea30c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:32:51.439 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:51.441 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:51.442 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:51.442 INFO 2458 --- [ocp-schedules-15,1d785c250841627d,a793453be12bbd9d] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:32:51.446 INFO 2458 --- [host-check-clockdiff1,ecfc974a08688f94,7aeb68cfef1175a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535971 0 0 , error= 2025-10-27 11:32:51.454 INFO 2458 --- [host-check-clockdiff2,ecfc974a08688f94,41028fc8d87fa643] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535971 -1 -1 , error= 2025-10-27 11:32:51.459 INFO 2458 --- [host-check-clockdiff5,ecfc974a08688f94,71d901e73bea30c5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535971 0 0 , error= 2025-10-27 11:32:51.473 INFO 2458 --- [host-check-clockdiff3,ecfc974a08688f94,8753058778b8e232] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535971 0 0 , error= 2025-10-27 11:32:51.507 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:32:51.507 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:32:51.510 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:32:51.510234515+08:00 2025-10-27 11:32:51.511 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:32:51.510+08:00 2025-10-27 11:32:51.513 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:32:51.514 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:32:51.513877455+08:00 2025-10-27 11:32:51.516 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:32:51.517 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:32:51.517 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:32:51.517 INFO 2458 --- [host-check-agent-status3,a20598a15f8c31bb,e3eb46e4a49c7a67] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:32:51.518 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:32:51.514+08:00 2025-10-27 11:32:51.519 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:32:51.520 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:32:51.520 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:32:51.520 INFO 2458 --- [host-check-agent-status5,a20598a15f8c31bb,3dbc09c33096ae76] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-agent-status1,a20598a15f8c31bb,f296359141b11398] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:32:51.522 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:32:51.522665756+08:00 2025-10-27 11:32:51.523 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:32:51.522+08:00 2025-10-27 11:32:51.524 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:32:51.525 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:32:51.510+08:00 and status AVAILABLE 2025-10-27 11:32:51.526 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status4,a20598a15f8c31bb,594eedf262c51ca7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-agent-status2,a20598a15f8c31bb,ff57852a04d36a55] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:32:51.527795202+08:00 2025-10-27 11:32:51.528 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:32:51.528+08:00 2025-10-27 11:32:51.530 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:32:51.534 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:32:51.514+08:00 and status AVAILABLE 2025-10-27 11:32:51.536 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:32:51.537 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:32:51.536508097+08:00 2025-10-27 11:32:51.538 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:32:51.537+08:00 2025-10-27 11:32:51.539 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:32:51.522+08:00 and status AVAILABLE 2025-10-27 11:32:51.539 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:32:51.528+08:00 and status AVAILABLE 2025-10-27 11:32:51.540 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:32:51.543 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:32:51.545 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:32:51.537+08:00 and status AVAILABLE 2025-10-27 11:32:51.545 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:32:51.545 INFO 2458 --- [ocp-schedules-11,0e327b1ded0d0674,6ba008eaee1e070b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:32:51.549 INFO 2458 --- [ocp-schedules-9,4da2bc248595e482,31e6e4ac2edb31c7] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:32:51.549 INFO 2458 --- [ocp-schedules-11,0e327b1ded0d0674,6ba008eaee1e070b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:51.549 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:32:51.553 INFO 2458 --- [ocp-schedules-9,4da2bc248595e482,31e6e4ac2edb31c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:51.553 INFO 2458 --- [ocp-schedules-9,4da2bc248595e482,31e6e4ac2edb31c7] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:32:51.553 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:51.554 INFO 2458 --- [ocp-schedules-28,5187eba754ea6992,c373b53fb135e982] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:32:51.557 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:51.560 INFO 2458 --- [ocp-schedules-28,5187eba754ea6992,c373b53fb135e982] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:51.565 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:51.567 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:32:51.567 INFO 2458 --- [ocp-schedules-13,d08d439b24ba8330,b4f2b5bf85651c08] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:32:51.568 INFO 2458 --- [ocp-schedules-21,85d08e9b88dd0781,00d2b37de270c08a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:32:51.568 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:51.569 INFO 2458 --- [ocp-schedules-8,bbc11383b971980c,f038efed463aa7ad] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:32:51.569 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:32:51.570 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:32:51.570 INFO 2458 --- [ocp-schedules-30,99d9f1f0ff701b92,aa8a76cb9cf4a2f3] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:32:51.572 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:32:51.573 INFO 2458 --- [ocp-schedules-13,d08d439b24ba8330,b4f2b5bf85651c08] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:32:51.577 INFO 2458 --- [ocp-schedules-21,85d08e9b88dd0781,00d2b37de270c08a] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:32:51.580 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:32:51.581 INFO 2458 --- [ocp-schedules-8,bbc11383b971980c,f038efed463aa7ad] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:32:51.583 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:32:51.589 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:51.590 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:51.616 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.617 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:51.618 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:32:51.636 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.636 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:32:51.639 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.640 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:51.643 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:51.644 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:51.645 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,d081ec1eb13c42dd,354912b617c3c8e4] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=d081ec1eb13c42dd, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:32:51.645 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.645 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,d081ec1eb13c42dd,354912b617c3c8e4] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=d081ec1eb13c42dd, duration=1 ms] 2025-10-27 11:32:51.646 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:51.646 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.647 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:32:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:32:51.652 INFO 2458 --- [ocp-schedules-11,0e327b1ded0d0674,6ba008eaee1e070b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:32:51.655 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:51.656 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:51.658 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:32:51.662 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.665 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:32:51.670 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:32:51.673 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:32:51.673 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.673 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:32:51.675 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:32:51.676 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:32:51.693 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:32:51.703 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,69b78be83c8f3213,24f4f8aaea6efee0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=69b78be83c8f3213, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:32:51.703 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,69b78be83c8f3213,24f4f8aaea6efee0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=69b78be83c8f3213, duration=0 ms] 2025-10-27 11:32:51.706 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.707 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:32:51.709 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:32:51.713 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:32:51.735 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:32:51.733790+08:00, lastProcessTime=2025-10-27T11:32:41.717289+08:00 2025-10-27 11:32:51.741 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:51.741 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.741 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:32:51.741 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.742 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:32:51.744 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.744 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:32:51.744 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:51.745 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.748 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:32:51.750 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.753 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:32:51.754 INFO 2458 --- [ocp-schedules-13,d08d439b24ba8330,b4f2b5bf85651c08] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:32:51.758 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:32:51.763 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:32:51.765 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:51.770 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:51.774 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:51.779 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:32:51.786 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:51.791 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:32:51.801 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:32:51.803 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:32:51.804 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:32:51.809 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:51.810 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.811 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:51.812 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.815 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:32:51.815 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:32:51.816 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:51.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:51.818 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.819 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:32:51.822 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:32:51.824 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:32:51.837 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:32:51.842 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:32:51.849 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:51.851 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:51.851 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:51.851 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:51.861 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:32:51.864 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:32:51.866 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:51.871 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:51.872 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:32:51.879 INFO 2458 --- [ocp-schedules-18,fbec139dc02c0c00,5f2d07e6896cff1f] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:32:51.879 INFO 2458 --- [ocp-schedules-18,fbec139dc02c0c00,5f2d07e6896cff1f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:32:51.884 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:51.887 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:32:51.889 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:32:51.895 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:32:51.897 INFO 2458 --- [ocp-schedules-2,fb9ccc6af847116f,b6912b260ac1f18c] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:32:51.903 INFO 2458 --- [host-check-host-status4,bc11b5c9171d9e48,5d9ea32c773f6a13] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:32:51.906 INFO 2458 --- [host-check-host-status3,bc11b5c9171d9e48,3b9d2dbd10bbbe04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:32:51.910 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:32:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:51.913 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.914 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:32:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:51.917 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:51.918 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:32:51.918 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:32:51.920 INFO 2458 --- [host-check-host-status5,bc11b5c9171d9e48,83d17fbf693fa8da] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:32:51.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:51.935 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:32:51.935 INFO 2458 --- [host-check-host-status1,bc11b5c9171d9e48,891a615f2490e64f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:32:51.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:32:51.947 INFO 2458 --- [host-check-host-status2,bc11b5c9171d9e48,a1ab4ce7008cbbb2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:32:51.948 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:32:51.967 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:51.967 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:51.985 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:32:51.733790+08:00 2025-10-27 11:32:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:52.007 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:32:52.039 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:52.039 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:52.049 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:32:52.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.051 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:32:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.057 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:32:52.072 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:32:52.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:52.078 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:32:52.094 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.098 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.099 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:32:52.110 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:52.110 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:32:52.121 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:32:52.121 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:32:52.123 INFO 2458 --- [pool-overview-container-1,6cb048535e043214,49cd906abcc38312] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:32:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.189 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.190 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:32:52.193 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:32:52.203 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:32:52.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:32:52.218 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:32:52.240 INFO 2458 --- [pool-overview-container-7,995da4b8ef592d42,f299d8e083446dab] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.260 INFO 2458 --- [pool-overview-container-4,5629b557225c1e58,8e99b6915192b7fe] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.265 INFO 2458 --- [pool-overview-container-4,5629b557225c1e58,8e99b6915192b7fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.267 INFO 2458 --- [pool-overview-container-4,5629b557225c1e58,8e99b6915192b7fe] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:32:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:32:52.292 INFO 2458 --- [pool-12-thread-5,995da4b8ef592d42,e6e43152d1662042] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:32:52.292 INFO 2458 --- [pool-12-thread-6,995da4b8ef592d42,6c9c2384b756488c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:32:52.298 INFO 2458 --- [pool-12-thread-7,995da4b8ef592d42,883db1106d4a90b4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:32:52.302 INFO 2458 --- [ForkJoinPool-7-worker-10,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.303 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:32:52.303 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:32:52.304 INFO 2458 --- [ForkJoinPool-7-worker-3,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.305 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:32:52.305 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:32:52.308 INFO 2458 --- [pool-12-thread-7,995da4b8ef592d42,883db1106d4a90b4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:32:52.308 INFO 2458 --- [pool-12-thread-7,995da4b8ef592d42,883db1106d4a90b4] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:32:52.313 INFO 2458 --- [pool-12-thread-8,995da4b8ef592d42,8a70c75783031f5c] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:32:52.318 INFO 2458 --- [pool-12-thread-5,995da4b8ef592d42,e6e43152d1662042] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:32:52.324 INFO 2458 --- [ForkJoinPool-7-worker-5,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.325 INFO 2458 --- [ForkJoinPool-7-worker-12,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:32:52.346 INFO 2458 --- [pool-12-thread-5,995da4b8ef592d42,e6e43152d1662042] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:32:52.347 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:32:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.348 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:32:52.349 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.350 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.350 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:32:52.351 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:32:52.351 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.351 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:32:52.352 INFO 2458 --- [pool-12-thread-5,995da4b8ef592d42,e6e43152d1662042] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:32:52.370 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.370 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:32:52.372 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.374 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:32:52.374 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.375 INFO 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:32:52.381 INFO 2458 --- [host-check-clockdiff4,ecfc974a08688f94,3ed9a28a85001b31] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761535972 0 0 , error= 2025-10-27 11:32:52.390 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:32:52.390 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.391 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:32:52.392 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.393 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:32:52.393 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:32:52.396 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.399 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:32:52.403 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:32:52.404 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.405 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:32:52.406 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.407 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:32:52.409 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.413 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:32:52.418 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:32:52.425 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.426 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:32:52, 2025-10-27 11:32:52] 2025-10-27 11:32:52.430 WARN 2458 --- [ForkJoinPool-7-worker-3,995da4b8ef592d42,21b3f5073fea2959] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:32:52.435 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.435 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.436 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:32:52.450 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:32:52.452 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.453 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:32:52.454 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.455 INFO 2458 --- [ForkJoinPool-7-worker-5,995da4b8ef592d42,af772995043235c2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:32:52.462 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:32:52.463 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.464 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:32:52, 2025-10-27 11:32:52] 2025-10-27 11:32:52.466 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:32:52.466 INFO 2458 --- [ForkJoinPool-7-worker-10,995da4b8ef592d42,36259b4f6fde3530] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:32:52.466 INFO 2458 --- [pool-12-thread-6,995da4b8ef592d42,6c9c2384b756488c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:32:52.474 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.477 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:32:52.479 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.483 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:32:52.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:32:52.489 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:32:52.489 INFO 2458 --- [ocp-schedules-28,5187eba754ea6992,c373b53fb135e982] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:32:52.499 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.501 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:32:52.501 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.502 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:32:52.503 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.504 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.504 INFO 2458 --- [ForkJoinPool-7-worker-12,995da4b8ef592d42,58e4619f170acfa9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:32:52.510 INFO 2458 --- [pool-12-thread-8,995da4b8ef592d42,8a70c75783031f5c] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:32:52.511 INFO 2458 --- [refresh-tenant-resource-cache-10,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:32:52.605 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:32:52.606 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:32:52.610 INFO 2458 --- [refresh-tenant-resource-cache-9,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:32:52.639 WARN 2458 --- [ocp-async-1,2856e023a6873a4b,f60586057a34c9d0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:52.639 INFO 2458 --- [ocp-async-6,e08bc5f74ed3e113,547e2ace94381055] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:52.825 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,71b3327328eaa0d0,408013bcc2acd890] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=71b3327328eaa0d0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:32:52.826 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,71b3327328eaa0d0,408013bcc2acd890] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=71b3327328eaa0d0, duration=1 ms] 2025-10-27 11:32:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:32:53.707 INFO 2458 --- [prometheus-minute-schedule-1,252485de39d3ca9e,6f0b739e951909f9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:53.715 INFO 2458 --- [prometheus-minute-schedule-1,252485de39d3ca9e,6f0b739e951909f9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:54.172 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.185 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=n 2025-10-27 11:32:54.187 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.187 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:32:54.188 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.189 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.197 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.201 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.216 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.218 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.221 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.223 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.224 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.271 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:32:54.441 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.443 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.443 INFO 2458 --- [ocp-schedules-26,065b2af06c45f260,b53032a6d6fb698a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:32:54.445 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.454 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.456 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.457 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.459 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.461 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.462 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.469 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.469 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:32:54.471 WARN 2458 --- [scanner-112206805-1,065b2af06c45f260,5f759c55a3f6434f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:32:54.473 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.477 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.496 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.497 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.499 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.501 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.502 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.503 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:26:53.229, end=2025-10-27T11:31:54.167} 2025-10-27 11:32:54.504 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:32:54.590 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:54.591 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:54.639 WARN 2458 --- [ocp-async-9,e72bca66c38a927d,6807ff18467bb14a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:54.639 INFO 2458 --- [ocp-async-2,b1e3700d343bb723,d997bdc1acaacceb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:55.612 INFO 2458 --- [ocp-schedules-20,6a0cf90d55aa5a44,b4bd7aa24beb586d] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:32:55.651 INFO 2458 --- [ocp-schedules-18,f148b374faf470b6,6a075c12ba875fbc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:32:55.654 INFO 2458 --- [ocp-schedules-18,f148b374faf470b6,6a075c12ba875fbc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:32:56.639 WARN 2458 --- [ocp-async-4,06b3c2c729420103,f5a37cd00b4b1ff6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:56.639 INFO 2458 --- [ocp-async-10,2dc2a99e6e5e0c9d,32bdb1e94342d679] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:56.707 INFO 2458 --- [prometheus-minute-schedule-1,8f9461747af72c55,9f48a796f8a51ff5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:56.717 INFO 2458 --- [prometheus-minute-schedule-1,8f9461747af72c55,9f48a796f8a51ff5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:32:57.031 INFO 2458 --- [ocp-schedules-5,83fe6f38fbad730c,144c4e54d2aebf3b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:32:57.591 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:32:57.592 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:32:58.540 INFO 2458 --- [ocp-async-10,6bd524537358dd22,858417d16b80f172] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:32:58.639 INFO 2458 --- [ocp-async-7,78d174f78fb3a676,21d6e5be09e5ab1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:32:58.639 WARN 2458 --- [ocp-async-6,f48c2d1b04488c59,64ee98445e4e6417] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:32:59.527 INFO 2458 --- [ocp-schedules-16,edfd18ece58e3fe1,c776ab77512a7e51] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:32:59.707 INFO 2458 --- [prometheus-minute-schedule-1,8b8887ce14c9a9a9,3b87d2e956df9455] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761535980, interval=60 2025-10-27 11:32:59.716 INFO 2458 --- [prometheus-minute-schedule-1,8b8887ce14c9a9a9,3b87d2e956df9455] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:00.592 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:00.593 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:00.639 WARN 2458 --- [ocp-async-2,8dad1737302d7a68,0050bc8c4d79d8ec] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:00.639 INFO 2458 --- [ocp-async-9,b8faf6e43c66e965,e858f7b4b9db556a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:01.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:01.720263+08:00, lastProcessTime=2025-10-27T11:32:51.733790+08:00 2025-10-27 11:33:02.264 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:01.720263+08:00 2025-10-27 11:33:02.640 INFO 2458 --- [ocp-async-9,e8153916aa629c55,553dcfb2035d1aff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:02.640 WARN 2458 --- [ocp-async-3,3d3dde7a30bf8551,7ec6d925a0a5f29a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:02.707 INFO 2458 --- [prometheus-minute-schedule-1,ea9ec6b315c97da4,6137e974cfe3a4a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:02.718 INFO 2458 --- [prometheus-minute-schedule-1,ea9ec6b315c97da4,6137e974cfe3a4a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:03.593 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:03.594 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:04.581 INFO 2458 --- [ocp-schedules-26,be0d36b333759930,beb81dce30902de0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:04.624 INFO 2458 --- [ocp-schedules-23,471a3e6bca261af2,87bf474eb1d8e41c] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:33:04.629 INFO 2458 --- [ocp-schedules-23,471a3e6bca261af2,87bf474eb1d8e41c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:04.640 INFO 2458 --- [ocp-async-6,cc487b90a0df893f,b89ff447f3bab378] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:04.640 WARN 2458 --- [ocp-async-1,5c3e71d5adfea4c9,d070803e6a3f5185] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:04.650 WARN 2458 --- [scanner-112207013-1,be0d36b333759930,75558b4c65d125a8] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:04.712 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:04.715 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:33:04.723 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:04.724 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:33:04.763 INFO 2458 --- [ocp-schedules-23,471a3e6bca261af2,87bf474eb1d8e41c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:33:04.811 INFO 2458 --- [alarm-detect-14,be0d36b333759930,9391187ae2a5e80c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535950 ~ 1761535974 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535965, value=59.396)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535965, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535965=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:33:05.662 INFO 2458 --- [ocp-schedules-5,c13ddf2e1d395137,c39f4a44a555b4b9] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:05.666 INFO 2458 --- [ocp-schedules-5,c13ddf2e1d395137,c39f4a44a555b4b9] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:05.707 INFO 2458 --- [prometheus-minute-schedule-1,b54d638437ca54fe,dcfc2079b6687f35] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:05.716 INFO 2458 --- [prometheus-minute-schedule-1,b54d638437ca54fe,dcfc2079b6687f35] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:06.594 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:06.595 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:06.640 INFO 2458 --- [ocp-async-6,5050bf7978eccf6d,4682649efe063179] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:06.640 WARN 2458 --- [ocp-async-8,0302350ea527af24,1c4efdd4800fa931] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:07.043 INFO 2458 --- [ocp-schedules-19,0f6453f7a1e279ac,3436fa3da857f9fc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:07.686 INFO 2458 --- [ocp-schedules-5,d7f3a855d0b5dad5,3e456473e7b0a452] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:33:07.695 INFO 2458 --- [ocp-schedules-5,d7f3a855d0b5dad5,3e456473e7b0a452] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:07.695 INFO 2458 --- [ocp-schedules-5,d7f3a855d0b5dad5,3e456473e7b0a452] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:08.540 INFO 2458 --- [ocp-async-2,d6689be51e237da2,963a77086156ff35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:08.640 WARN 2458 --- [ocp-async-9,b5b1f82a53954fb7,4bcc47653a90a3f1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:08.640 INFO 2458 --- [ocp-async-4,783595aa4c472230,881cd6aee08ccfd0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:08.707 INFO 2458 --- [prometheus-minute-schedule-1,adacab4612060420,3cf0ea40d5bb5363] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:08.715 INFO 2458 --- [prometheus-minute-schedule-1,adacab4612060420,3cf0ea40d5bb5363] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:09.595 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:09.596 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:09.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,509a953d2b84d484,694d1a6ff60a85ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=509a953d2b84d484, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:33:09.705 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,509a953d2b84d484,694d1a6ff60a85ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=509a953d2b84d484, duration=1 ms] 2025-10-27 11:33:10.011 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:33:10.014 INFO 2458 --- [ocp-schedules-15,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:33:10.061 INFO 2458 --- [Check obproxy1,8940e3c5b8590016,7dde05bd337cccc7] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:33:10.061 INFO 2458 --- [Collect obproxy1,5a06d6dccf76a0ea,f9b1b3a6fdfba743] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:33:10.061 INFO 2458 --- [Check obproxy1,8940e3c5b8590016,7dde05bd337cccc7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:33:10.061 INFO 2458 --- [Collect obproxy1,5a06d6dccf76a0ea,f9b1b3a6fdfba743] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:33:10.093 INFO 2458 --- [Check obproxy1,8940e3c5b8590016,7dde05bd337cccc7] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:33:10.096 INFO 2458 --- [Collect obproxy1,5a06d6dccf76a0ea,f9b1b3a6fdfba743] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:33:10.097 INFO 2458 --- [Check obproxy1,8940e3c5b8590016,7dde05bd337cccc7] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:33:10.097 INFO 2458 --- [Check obproxy1,8940e3c5b8590016,7dde05bd337cccc7] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:33:10.099 INFO 2458 --- [Collect obproxy1,5a06d6dccf76a0ea,f9b1b3a6fdfba743] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:33:10.102 INFO 2458 --- [Collect obproxy1,5a06d6dccf76a0ea,f9b1b3a6fdfba743] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:33:10.181 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:33:10.640 INFO 2458 --- [ocp-async-7,35ddc591a786b63f,dacbca8c5123e27d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:10.640 WARN 2458 --- [ocp-async-5,432b0b1219e4ffcd,1e82c2c19075c9f5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:11.541 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=1000, affectRows=1000 2025-10-27 11:33:11.547 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=994, validCount=992 2025-10-27 11:33:11.550 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587948) Duplicate entry '835014129065861120-1761535860' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587948) Duplicate entry '835014129065861120-1761535860' for key 'PRIMARY' 2025-10-27 11:33:11.550 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505683) Duplicate entry '835370602979598336-1761535860' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505683) Duplicate entry '835370602979598336-1761535860' for key 'PRIMARY' 2025-10-27 11:33:11.561 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=1000, affectRows=1000 2025-10-27 11:33:11.565 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=999, validCount=995 2025-10-27 11:33:11.581 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=78, affectRows=78 2025-10-27 11:33:11.581 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=77, validCount=77 2025-10-27 11:33:11.613 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=1000, affectRows=1000 2025-10-27 11:33:11.617 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=996, validCount=992 2025-10-27 11:33:11.707 INFO 2458 --- [prometheus-minute-schedule-1,d95b0451e7c42be6,ea98c1d9f93d651e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:11.715 INFO 2458 --- [prometheus-minute-schedule-1,d95b0451e7c42be6,ea98c1d9f93d651e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:11.727 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:11.725906+08:00, lastProcessTime=2025-10-27T11:33:01.720263+08:00 2025-10-27 11:33:12.256 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:11.725906+08:00 2025-10-27 11:33:12.590 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535920, nodeCount=821, affectRows=821 2025-10-27 11:33:12.596 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:12.597 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:12.640 WARN 2458 --- [ocp-async-3,1557dfe37f88955b,c084006f4c7ac224] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:12.640 INFO 2458 --- [ocp-async-6,67ff610a1ac6c6f6,3b07dc042d4ef488] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:14.122 INFO 2458 --- [ocp-schedules-16,6bb9dc92d80a87a0,83fce72738f99ca9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:33:14.129 INFO 2458 --- [ocp-schedules-16,6bb9dc92d80a87a0,83fce72738f99ca9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:14.245 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:33:14.246 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.250 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:33:14.262 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.263 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.281 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.281 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.281 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:33:14.281 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.281 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.282 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.285 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:33:14.292 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.293 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.304 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.304 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.304 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.305 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.324 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.325 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.327 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.327 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.336 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.336 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.337 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.357 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.357 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.363 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.363 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.363 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.364 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:33:14.366 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.366 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.366 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.367 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.384 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.384 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.396 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.396 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.397 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.397 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.423 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.423 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.423 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.423 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:33:14.425 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.425 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.426 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.426 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:33:14.447 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.448 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.449 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.450 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.450 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.451 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.452 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.453 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.454 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.455 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.455 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.456 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.456 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:33:14.457 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.458 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.458 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.458 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:33:14.463 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:33:14.489 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.489 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.489 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.490 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:33:14.492 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.492 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.492 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.493 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:33:14.522 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.522 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.522 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.523 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:33:14.527 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:14.528 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.528 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:33:14.539 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.539 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:33:14.539 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.539 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:33:14.542 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:33:14.542 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:33:14.549 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.549 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.549 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.549 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:33:14.594 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.594 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.595 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.595 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.596 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.597 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.597 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.597 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.597 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.598 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.600 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.600 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.600 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.601 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.601 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.601 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.602 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.603 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.603 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.603 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.604 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.604 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.604 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.604 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.605 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.606 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.606 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.606 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.607 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.607 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.607 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.607 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.608 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.608 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.608 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.609 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.609 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:33:14.610 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.610 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.610 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.611 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:33:14.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:33:14.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:33:14.612 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.612 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:33:14.613 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:33:14.615 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:33:14.641 WARN 2458 --- [ocp-async-9,9bfb031d5a246f2d,f104dfcd81095600] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:14.641 INFO 2458 --- [ocp-async-9,f5aad42d7b95c71e,3d3fc7a06fd113ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:14.707 INFO 2458 --- [prometheus-minute-schedule-1,e554a8bb8ad02d31,251670fb89a53a5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:14.716 INFO 2458 --- [prometheus-minute-schedule-1,e554a8bb8ad02d31,251670fb89a53a5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:14.722 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:14.723 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:14.724 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:33:14.759 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:33:14.764 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:33:14.764 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:33:14.764 INFO 2458 --- [ocp-schedules-16,6bb9dc92d80a87a0,83fce72738f99ca9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:33:14.818 INFO 2458 --- [ocp-schedules-30,d3354f4acec9cb1c,3f293ff677568cab] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:14.874 WARN 2458 --- [scanner-112208833-1,d3354f4acec9cb1c,ff042c9d61cbc308] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:14.944 INFO 2458 --- [alarm-detect-14,d3354f4acec9cb1c,e37e67f365de460f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535965 ~ 1761535984 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535980, value=55.391)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535980, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535980=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:33:15.597 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:15.598 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:15.675 INFO 2458 --- [ocp-schedules-27,0d94273afad2ff85,3f8232d5370a6e77] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:15.680 INFO 2458 --- [ocp-schedules-27,0d94273afad2ff85,3f8232d5370a6e77] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:16.165 INFO 2458 --- [ocp-schedules-18,cd1eed0ceca7c20c,1e7ea46453397a63] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:33:16.165+08:00, clusters=[1, 2] 2025-10-27 11:33:16.168 INFO 2458 --- [ocp-schedules-18,cd1eed0ceca7c20c,1e7ea46453397a63] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:33:16.172 INFO 2458 --- [ocp-schedules-18,cd1eed0ceca7c20c,1e7ea46453397a63] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:33:16.416 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=1000, affectRows=1000 2025-10-27 11:33:16.420 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=1000, validCount=998 2025-10-27 11:33:16.485 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535920, nodeCount=1000, affectRows=1000 2025-10-27 11:33:16.489 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535920, nodeCount=996, validCount=982 2025-10-27 11:33:16.641 WARN 2458 --- [ocp-async-9,8d3d192030e9f190,bc5c8b51e6c6e6dc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:16.641 INFO 2458 --- [ocp-async-4,3ab8c3d3229fa548,02b3e72cec78e51a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:17.052 INFO 2458 --- [ocp-schedules-2,ad02325b2680f49b,e4d766e4a3909dd4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:17.707 INFO 2458 --- [prometheus-minute-schedule-1,c8418f9b9b58cdbc,bc22caacc779e051] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:17.714 INFO 2458 --- [prometheus-minute-schedule-1,c8418f9b9b58cdbc,bc22caacc779e051] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:18.074 INFO 2458 --- [ocp-schedules-4,ac86b61c769d07ec,f9892ce163461de1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:33:18.074 INFO 2458 --- [ocp-schedules-4,ac86b61c769d07ec,f9892ce163461de1] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:33:18.128 INFO 2458 --- [Collect all tenants major compactions2,ac86b61c769d07ec,825c466c72a63493] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:33:18.128 INFO 2458 --- [Collect all tenants major compactions1,ac86b61c769d07ec,cdeccad3e58faf22] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:33:18.129 INFO 2458 --- [Collect all tenants major compactions3,ac86b61c769d07ec,1a3c232199ca7b1e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:33:18.129 INFO 2458 --- [Collect all tenants major compactions4,ac86b61c769d07ec,e6c2c3fac0f64493] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:33:18.129 INFO 2458 --- [Collect all tenants major compactions5,ac86b61c769d07ec,12328e28ecd92c85] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:33:18.131 INFO 2458 --- [Collect all tenants major compactions6,ac86b61c769d07ec,602736375eb8048b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:33:18.167 INFO 2458 --- [Collect all tenants major compactions5,ac86b61c769d07ec,12328e28ecd92c85] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.168 INFO 2458 --- [Collect all tenants major compactions6,ac86b61c769d07ec,602736375eb8048b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.169 INFO 2458 --- [Collect all tenants major compactions6,ac86b61c769d07ec,602736375eb8048b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:33:18.169 INFO 2458 --- [Collect all tenants major compactions5,ac86b61c769d07ec,12328e28ecd92c85] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:33:18.178 INFO 2458 --- [Collect all tenants major compactions6,ac86b61c769d07ec,602736375eb8048b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:33:18.180 INFO 2458 --- [Collect all tenants major compactions5,ac86b61c769d07ec,12328e28ecd92c85] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:33:18.186 INFO 2458 --- [Collect all tenants major compactions4,ac86b61c769d07ec,e6c2c3fac0f64493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.186 INFO 2458 --- [Collect all tenants major compactions1,ac86b61c769d07ec,cdeccad3e58faf22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.186 INFO 2458 --- [Collect all tenants major compactions3,ac86b61c769d07ec,1a3c232199ca7b1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.187 INFO 2458 --- [Collect all tenants major compactions4,ac86b61c769d07ec,e6c2c3fac0f64493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:33:18.187 INFO 2458 --- [Collect all tenants major compactions1,ac86b61c769d07ec,cdeccad3e58faf22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:33:18.187 INFO 2458 --- [Collect all tenants major compactions3,ac86b61c769d07ec,1a3c232199ca7b1e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:33:18.191 INFO 2458 --- [Collect all tenants major compactions4,ac86b61c769d07ec,e6c2c3fac0f64493] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:33:18.191 INFO 2458 --- [Collect all tenants major compactions1,ac86b61c769d07ec,cdeccad3e58faf22] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:33:18.191 INFO 2458 --- [Collect all tenants major compactions3,ac86b61c769d07ec,1a3c232199ca7b1e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:33:18.199 INFO 2458 --- [Collect all tenants major compactions2,ac86b61c769d07ec,825c466c72a63493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:18.200 INFO 2458 --- [Collect all tenants major compactions2,ac86b61c769d07ec,825c466c72a63493] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:33:18.203 INFO 2458 --- [Collect all tenants major compactions2,ac86b61c769d07ec,825c466c72a63493] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:33:18.204 INFO 2458 --- [ocp-schedules-4,ac86b61c769d07ec,f9892ce163461de1] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:33:18.540 INFO 2458 --- [ocp-async-8,44b7cecba7a186c0,2b4d99f606622815] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:18.598 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:18.599 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:18.641 INFO 2458 --- [ocp-async-7,2201d1d2e4ea772e,691998c71a5386b8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:18.641 WARN 2458 --- [ocp-async-2,0a972c1863506019,ce6e768d363c50db] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:20.640 INFO 2458 --- [ocp-schedules-3,0a2f9dc371ecda65,e94f421f2367cf6a] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:33:20.641 WARN 2458 --- [ocp-async-1,cd75c7f12664650e,06b00ca86c8441a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:20.641 INFO 2458 --- [ocp-async-6,e2b3daead9396e45,36a0f2c30e10d0c0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:20.707 INFO 2458 --- [prometheus-minute-schedule-1,dc1000fdf16631ef,a6cb170eae0dfce7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:20.713 INFO 2458 --- [prometheus-minute-schedule-1,dc1000fdf16631ef,a6cb170eae0dfce7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:21.409 INFO 2458 --- [ocp-schedules-10,c2d43e84c727b1ce,8f34dffeb97b1f77] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:21.409 INFO 2458 --- [ocp-schedules-9,990620d7afd00cd0,60cdbf1ce4e9fa2b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:21.410 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:33:21.410 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:33:21.411 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:33:21.411 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:33:21.411 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:33:21.411 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:33:21.411 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:33:21.412 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:33:21.412 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:33:21.412 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:33:21.424 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:21.424 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:21.424 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:21.424 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:21.425 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:21.425 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:21.425 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:21.426 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:21.428 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:21.428 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:21.429 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:21.429 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:21.429 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:21.429 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:21.430 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:21.430 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:21.431 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:21.432 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:21.433 INFO 2458 --- [ocp-schedules-6,52db40871b784a94,eaaaeeba59737256] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:21.433 INFO 2458 --- [host-check-clockdiff1,52db40871b784a94,2a979ca6b730e564] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:33:21.434 INFO 2458 --- [host-check-clockdiff3,52db40871b784a94,b03db950e73dc74f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:33:21.434 INFO 2458 --- [host-check-clockdiff2,52db40871b784a94,7f593bf1eeb8582f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:33:21.434 INFO 2458 --- [host-check-clockdiff5,52db40871b784a94,3c399341bbe639dc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:33:21.434 INFO 2458 --- [host-check-clockdiff4,52db40871b784a94,528bbd5d723c70c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:33:21.434 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:21.435 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:21.442 INFO 2458 --- [ocp-schedules-23,0459bcef2c61f2df,4676ec68ba3b3bc3] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:33:21.491 INFO 2458 --- [host-check-clockdiff3,52db40871b784a94,b03db950e73dc74f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536001 0 0 , error= 2025-10-27 11:33:21.512 INFO 2458 --- [host-check-clockdiff4,52db40871b784a94,528bbd5d723c70c2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536001 0 0 , error= 2025-10-27 11:33:21.512 INFO 2458 --- [host-check-clockdiff5,52db40871b784a94,3c399341bbe639dc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536001 0 0 , error= 2025-10-27 11:33:21.514 INFO 2458 --- [host-check-clockdiff1,52db40871b784a94,2a979ca6b730e564] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536001 0 0 , error= 2025-10-27 11:33:21.546 INFO 2458 --- [ocp-schedules-15,0091e03aa54441bc,f1b4ab7f4ce79fd9] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:33:21.549 INFO 2458 --- [ocp-schedules-11,ee023b0e7373debf,a77d65b8ccd35b11] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:33:21.554 INFO 2458 --- [ocp-schedules-11,ee023b0e7373debf,a77d65b8ccd35b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:21.554 INFO 2458 --- [ocp-schedules-14,e72eca1fed5721ba,51df3f5298a1fd29] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:33:21.554 INFO 2458 --- [ocp-schedules-15,0091e03aa54441bc,f1b4ab7f4ce79fd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:21.558 INFO 2458 --- [ocp-schedules-14,e72eca1fed5721ba,51df3f5298a1fd29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:21.565 INFO 2458 --- [ocp-schedules-11,ee023b0e7373debf,a77d65b8ccd35b11] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:33:21.571 INFO 2458 --- [ocp-schedules-20,f4520edb98b99e89,4235d1f65a38f4ac] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:33:21.620 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:21.621 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:21.637 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:33:21.640 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:33:21.641 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:33:21.641556601+08:00 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-agent-status5,990620d7afd00cd0,1272148189d8d74b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:33:21.642 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:33:21.643 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:33:21.643041048+08:00 2025-10-27 11:33:21.643 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:33:21.642797819+08:00 2025-10-27 11:33:21.643 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:33:21.642+08:00 2025-10-27 11:33:21.644 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:33:21.643+08:00 2025-10-27 11:33:21.644 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:33:21.643+08:00 2025-10-27 11:33:21.644 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:33:21.643111174+08:00 2025-10-27 11:33:21.645 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:33:21.644+08:00 2025-10-27 11:33:21.655 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.656 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:21.657 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:33:21.643+08:00 and status AVAILABLE 2025-10-27 11:33:21.657 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:33:21.658 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:33:21.659 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:33:21.642+08:00 and status AVAILABLE 2025-10-27 11:33:21.660 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:33:21.643+08:00 and status AVAILABLE 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:33:21.644+08:00 and status AVAILABLE 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-agent-status3,990620d7afd00cd0,55a017fc7fd6437c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:33:21.661 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:33:21.662 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:33:21.662 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:33:21.662 INFO 2458 --- [host-check-agent-status4,990620d7afd00cd0,05df0619fb4bff4d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:33:21.665 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:33:21.668 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:33:21.668 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:33:21.683 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.683 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:21.684 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:33:21.684 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:21.687 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:33:21.687 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:33:21.687 INFO 2458 --- [host-check-agent-status2,990620d7afd00cd0,961c7b410ff3c3af] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:33:21.689 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:33:21.689 INFO 2458 --- [ocp-schedules-15,0091e03aa54441bc,f1b4ab7f4ce79fd9] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:33:21.692 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:33:21.693 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:33:21.693 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:33:21.693582065+08:00 2025-10-27 11:33:21.694 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:21.694 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:33:21.693+08:00 2025-10-27 11:33:21.696 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:33:21.696 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:21.696 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:33:21.696 INFO 2458 --- [host-check-agent-status1,990620d7afd00cd0,f826fbaa28a9995a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:33:21.697 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:21.699 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:21.702 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:21.704 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:21.707 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:33:21.693+08:00 and status AVAILABLE 2025-10-27 11:33:21.707 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:21.709 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:21.714 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:33:21.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:21.719015+08:00, lastProcessTime=2025-10-27T11:33:11.725906+08:00 2025-10-27 11:33:21.738 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:21.741 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:21.753 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.753 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:21.769 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:21.773 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:21.773 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:33:21.775 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:21.775 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:21.790 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:21.790 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:33:21.792 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:33:21.796 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:33:21.805 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:33:21.816 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:21.836 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:33:21.843 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:33:21.846 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:33:21.847 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:33:21.848 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:33:21.851 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:21.851 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.852 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:21.853 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.853 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.854 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:33:21.857 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:33:21.858 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:21.873 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:21.875 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:33:21.877 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:21.877 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:21.881 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:21.881 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:21.890 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:33:21.896 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:33:21.898 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:33:21.900 INFO 2458 --- [ocp-schedules-3,2b178f87e7815167,31527e5d026722dd] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:33:21.902 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:33:21.904 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:21.924 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:33:21.932 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:33:21.933 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:33:21.936 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:33:21.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:21.940 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.941 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:21.941 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:33:21.942 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:33:21.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:21.948 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.949 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:21.950 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:21.953 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:33:21.953 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:33:21.955 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:21.961 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:21.964 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:21.964 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:21.966 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:21.967 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:21.967 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:21.972 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:33:21.978 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:33:21.982 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:33:21.987 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:33:21.989 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:21.719015+08:00 2025-10-27 11:33:21.992 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:33:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:21.996 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:22.001 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:22.016 INFO 2458 --- [host-check-host-status5,c2d43e84c727b1ce,b58cf9569bb4a66c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:33:22.022 INFO 2458 --- [host-check-host-status4,c2d43e84c727b1ce,4bc458cc938ff94f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:33:22.029 INFO 2458 --- [host-check-host-status2,c2d43e84c727b1ce,c488cb1f60aba52e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:33:22.032 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:33:22.035 INFO 2458 --- [host-check-host-status3,c2d43e84c727b1ce,4abc2c661789952b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:33:22.039 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:33:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:22.054 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:22.058 INFO 2458 --- [host-check-host-status1,c2d43e84c727b1ce,cf1d14ee697949c1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:33:22.084 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:33:22.084 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.085 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:33:22.085 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:22.085 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:22.087 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.088 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:33:22.102 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:33:22.126 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:22.126 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:22.126 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:22.127 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.127 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:33:22.150 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:33:22.150 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:33:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:22.205 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.246 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:22.249 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:33:22.258 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:33:22.267 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:22.270 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:22.318 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:33:22.383 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:33:22.383 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.384 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:33:22.392 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:33:22.426 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:33:22.464 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:22.465 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:22.466 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:33:22.505 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:33:22.505 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:33:22.505 INFO 2458 --- [ocp-schedules-14,e72eca1fed5721ba,51df3f5298a1fd29] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:33:22.641 INFO 2458 --- [ocp-async-9,574ee6e3e17da453,3d72b030928e93c7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:22.641 WARN 2458 --- [ocp-async-8,5a262aae2c39b8e0,4015243d0386213b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:23.231 INFO 2458 --- [host-check-clockdiff2,52db40871b784a94,7f593bf1eeb8582f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536003 -1 -1 , error= 2025-10-27 11:33:23.707 INFO 2458 --- [prometheus-minute-schedule-1,3aa0e714abefb70e,683b6ec3bddb0921] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:23.714 INFO 2458 --- [prometheus-minute-schedule-1,3aa0e714abefb70e,683b6ec3bddb0921] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:24.621 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:24.622 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:24.641 INFO 2458 --- [ocp-async-4,5de59bada4626afb,dfb4c0cdf984e20e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:24.641 WARN 2458 --- [ocp-async-3,3d33c9ab3e2aec8c,5df6d045f5db98c1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:24.951 INFO 2458 --- [ocp-schedules-7,59e542da485473fc,2ff7d08a145445de] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:24.996 WARN 2458 --- [scanner-112208997-1,59e542da485473fc,0d6370f2defc0b8b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:25.149 INFO 2458 --- [alarm-detect-19,59e542da485473fc,b60b78c7df3e9df1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535980 ~ 1761535995 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535995, value=60.476)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535995, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761535995=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:33:25.693 INFO 2458 --- [ocp-schedules-30,98197c528cac59f6,a498ce5655690fe1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:25.696 INFO 2458 --- [ocp-schedules-30,98197c528cac59f6,a498ce5655690fe1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:26.641 WARN 2458 --- [ocp-async-1,1c845edd7535d321,dfdec1b68cd47673] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:26.641 INFO 2458 --- [ocp-async-7,38f3dd4dce3bb3fb,adf5bab14336279b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:26.707 INFO 2458 --- [prometheus-minute-schedule-1,9dd91112df5caba9,f8598c9f1e87c2c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:26.714 INFO 2458 --- [prometheus-minute-schedule-1,9dd91112df5caba9,f8598c9f1e87c2c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:27.059 INFO 2458 --- [ocp-schedules-21,1df5a4bcebbefef6,d4bbf1c6d4d40be2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:27.622 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:27.623 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:27.697 INFO 2458 --- [ocp-schedules-18,a4628c59ea26f97e,706938e5fb290394] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:33:27.705 INFO 2458 --- [ocp-schedules-18,a4628c59ea26f97e,706938e5fb290394] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:27.705 INFO 2458 --- [ocp-schedules-18,a4628c59ea26f97e,706938e5fb290394] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:28.540 INFO 2458 --- [ocp-async-7,dd96591176f7afd3,7dd0a60579dfc6aa] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:28.642 INFO 2458 --- [ocp-async-8,83ea80ea5d230111,178a6a8eaab24d11] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:28.642 WARN 2458 --- [ocp-async-9,f71a1e40edd1610c,2447d0cb26e5594a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:29.056 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:33:29.056 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:33:29.061 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:33:29.061 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:33:29.065 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:33:29.065 INFO 2458 --- [ocp-schedules-15,7ffc3428d359cfde,d08281fa9bf7e1bf] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:33:29.502 INFO 2458 --- [ocp-schedules-22,bd3a8e9749d178d4,82d79337cb9bb192] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:33:29.509 INFO 2458 --- [ocp-schedules-22,bd3a8e9749d178d4,82d79337cb9bb192] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:33:29.509 INFO 2458 --- [ocp-schedules-22,bd3a8e9749d178d4,82d79337cb9bb192] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:33:29.707 INFO 2458 --- [prometheus-minute-schedule-1,d26686d2e633e7ab,f49ecf7da188c03b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:29.715 INFO 2458 --- [prometheus-minute-schedule-1,d26686d2e633e7ab,f49ecf7da188c03b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:30.623 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:30.624 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:30.642 INFO 2458 --- [ocp-async-3,0dab42ef9fbd062d,131ac618b2016bff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:30.642 WARN 2458 --- [ocp-async-4,b0571ec0042bfac6,345253fd58fc8940] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:30.749 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6b31a1ba8bbcae61,3ea3c5672bccedb0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/binlog/clustersView?page=1&size=10, client=10.40.198.13, traceId=6b31a1ba8bbcae61, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.BinlogController.listBinlogClustersView(String,Pageable), args=Page request [number: 0, size 10, sort: name: ASC],] 2025-10-27 11:33:30.756 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,6b31a1ba8bbcae61,3ea3c5672bccedb0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/binlog/clustersView client=10.40.198.13, traceId=6b31a1ba8bbcae61, duration=21 ms] 2025-10-27 11:33:31.664 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d25489dc2941e8c8,56bdda1f6129f5f1] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=d25489dc2941e8c8, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:33:31.673 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d25489dc2941e8c8,56bdda1f6129f5f1] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=d25489dc2941e8c8, duration=18 ms] 2025-10-27 11:33:31.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:31.717293+08:00, lastProcessTime=2025-10-27T11:33:21.719015+08:00 2025-10-27 11:33:31.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:31.717293+08:00 2025-10-27 11:33:32.529 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,099ecd7d8a836217,cd8691d309a0cc0e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/profiles/me?null, client=10.40.198.13, traceId=099ecd7d8a836217, method=SuccessResponse com.oceanbase.ocp.server.common.controller.ProfileController.userInfo(), args=] 2025-10-27 11:33:32.529 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,a3a1da722b45ac26,4c563a2291c762f2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/config/systemInfo?null, client=10.40.198.13, traceId=a3a1da722b45ac26, method=SuccessResponse com.oceanbase.ocp.server.common.controller.PropertyController.getSystemInfo(), args=] 2025-10-27 11:33:32.529 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,a3a1da722b45ac26,4c563a2291c762f2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/config/systemInfo client=10.40.198.13, traceId=a3a1da722b45ac26, duration=6 ms] 2025-10-27 11:33:32.542 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,ee15744cc27e05da,5826f37f5030b91b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/alarm/alarms/actived?null, client=10.40.198.13, traceId=ee15744cc27e05da, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.AlarmController.fetchActivedAlarms(String,String,List,String,String,List,List,List,OffsetDateTime,OffsetDateTime,String,Pageable), args=Page request [number: 0, size 10, sort: id: DESC],] 2025-10-27 11:33:32.544 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,099ecd7d8a836217,cd8691d309a0cc0e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/profiles/me client=10.40.198.13, traceId=099ecd7d8a836217, duration=21 ms] 2025-10-27 11:33:32.544 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,44558375403fb0fc,c250d50ea4107b5e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/tasks/instances/state-statistics?null, client=10.40.198.13, traceId=44558375403fb0fc, method=IterableResponse com.oceanbase.ocp.server.common.controller.TaskController.getTaskStateSummary(TaskType), args=MANUAL,] 2025-10-27 11:33:32.545 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,44558375403fb0fc,c250d50ea4107b5e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/tasks/instances/state-statistics client=10.40.198.13, traceId=44558375403fb0fc, duration=18 ms] 2025-10-27 11:33:32.637 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,807cdae171a8f3c8,070d0a2068b0916f] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/telemetry?null, client=10.40.198.13, traceId=807cdae171a8f3c8, method=SuccessResponse com.oceanbase.ocp.controller.TelemetryController.get(), args=] 2025-10-27 11:33:32.642 INFO 2458 --- [ocp-async-10,fee7704480844413,446cf80bf6190a02] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:32.642 WARN 2458 --- [ocp-async-7,707230fd2b307706,7fad377625c2c817] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:32.677 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,807cdae171a8f3c8,070d0a2068b0916f] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/telemetry client=10.40.198.13, traceId=807cdae171a8f3c8, duration=47 ms] 2025-10-27 11:33:32.704 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:33:32.704 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/mc/info, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:33:32.704 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:33:32.707 INFO 2458 --- [prometheus-minute-schedule-1,92923862bbe08df4,33fe773938cc7194] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:32.713 INFO 2458 --- [prometheus-minute-schedule-1,92923862bbe08df4,33fe773938cc7194] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:32.761 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,eeb0b2974f92b1fb,13a6ad56b3429794] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.40.198.13, traceId=eeb0b2974f92b1fb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:33:32.762 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,7fffe8b5e7cb3f41,cc2b57a6fc7f652a] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?mode=MYSQL, client=10.40.198.13, traceId=7fffe8b5e7cb3f41, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],[MYSQL],NONE,] 2025-10-27 11:33:32.762 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3975aff1e0de1dc6,c0c96d35f10c08f0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.40.198.13, traceId=3975aff1e0de1dc6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-10-27 11:33:32.767 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,69c4c751c8c69beb,d5553c0aa087e690] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/software-packages?type=BINLOG_SERVICE_INSTALL_PACKAGE&withInstallationData=true, client=10.40.198.13, traceId=69c4c751c8c69beb, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.listPackages(String,List,List,List,List,boolean,Pageable), args=[BINLOG_SERVICE_INSTALL_PACKAGE],true,Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:33:32.770 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,9c316e2a5ff031da,88dc3c222c6a06f0] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?status=RUNNING, client=10.40.198.13, traceId=9c316e2a5ff031da, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=[RUNNING],false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-10-27 11:33:32.793 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3975aff1e0de1dc6,c0c96d35f10c08f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.40.198.13, traceId=3975aff1e0de1dc6, duration=40 ms] 2025-10-27 11:33:32.794 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,eeb0b2974f92b1fb,13a6ad56b3429794] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.40.198.13, traceId=eeb0b2974f92b1fb, duration=41 ms] 2025-10-27 11:33:32.811 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,9c316e2a5ff031da,88dc3c222c6a06f0] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.40.198.13, traceId=9c316e2a5ff031da, duration=58 ms] 2025-10-27 11:33:32.821 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,7fffe8b5e7cb3f41,cc2b57a6fc7f652a] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.40.198.13, traceId=7fffe8b5e7cb3f41, duration=68 ms] 2025-10-27 11:33:32.828 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,ee15744cc27e05da,5826f37f5030b91b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/alarm/alarms/actived client=10.40.198.13, traceId=ee15744cc27e05da, duration=299 ms] 2025-10-27 11:33:32.829 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,69c4c751c8c69beb,d5553c0aa087e690] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/software-packages client=10.40.198.13, traceId=69c4c751c8c69beb, duration=76 ms] 2025-10-27 11:33:32.921 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c0fc785c99f9c8d8,915a19eac0a5245d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.40.198.13, traceId=c0fc785c99f9c8d8, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:33:32.925 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,c0fc785c99f9c8d8,915a19eac0a5245d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.40.198.13, traceId=c0fc785c99f9c8d8, duration=10 ms] 2025-10-27 11:33:32.926 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,2fc5efeecf87a0fe,3e3b7580a02aa214] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hosts?status=AVAILABLE, client=10.40.198.13, traceId=2fc5efeecf87a0fe, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHosts(List,String,List,List,List,List,List,List,List,List,Pageable), args=[AVAILABLE],Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:33:32.926 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,63b01443516a0a8d,0b83531102e3130b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/tenants?withRemote=&mode=MYSQL, client=10.40.198.13, traceId=63b01443516a0a8d, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObTenantController.listAllTenants(Pageable,String,List,Boolean,String,Boolean,List,WithRemoteTenant,List), args=Page request [number: 0, size 5000, sort: id: DESC],[MYSQL],NONE,] 2025-10-27 11:33:32.928 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-8,2fc5efeecf87a0fe,3e3b7580a02aa214] o.h.h.internal.ast.QueryTranslatorImpl : HHH000104: firstResult/maxResults specified with collection fetch; applying in memory! 2025-10-27 11:33:32.928 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,e04c90ac7dca9ab1,e429ab9784de0b63] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/ob/clusters?null, client=10.40.198.13, traceId=e04c90ac7dca9ab1, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ObClusterController.listObClusters(String,List,boolean,Boolean,WithRemote,List,Pageable), args=false,NONE,Page request [number: 0, size 5000, sort: name: ASC,obClusterId: ASC],] 2025-10-27 11:33:32.931 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7a91cb47ceef73c6,3ec45bc172d6eecf] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?null, client=10.40.198.13, traceId=7a91cb47ceef73c6, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-10-27 11:33:32.933 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3d90e1f66b1c0a70,d5ce74b5faaaec46] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/idcs?status=AVAILABLE, client=10.40.198.13, traceId=3d90e1f66b1c0a70, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listIdcs(Pageable,Boolean), args=Page request [number: 0, size 5000, sort: id: DESC],false,] 2025-10-27 11:33:32.936 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,7a91cb47ceef73c6,3ec45bc172d6eecf] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.40.198.13, traceId=7a91cb47ceef73c6, duration=22 ms] 2025-10-27 11:33:32.938 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,63b01443516a0a8d,0b83531102e3130b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/tenants client=10.40.198.13, traceId=63b01443516a0a8d, duration=24 ms] 2025-10-27 11:33:32.939 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,3d90e1f66b1c0a70,d5ce74b5faaaec46] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/idcs client=10.40.198.13, traceId=3d90e1f66b1c0a70, duration=25 ms] 2025-10-27 11:33:32.957 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,e04c90ac7dca9ab1,e429ab9784de0b63] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/ob/clusters client=10.40.198.13, traceId=e04c90ac7dca9ab1, duration=43 ms] 2025-10-27 11:33:32.984 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,2fc5efeecf87a0fe,3e3b7580a02aa214] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hosts client=10.40.198.13, traceId=2fc5efeecf87a0fe, duration=70 ms] 2025-10-27 11:33:33.005 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,068e76362589f1ee,4a9ee3ff30c8d5ff] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/compute/hostTypes?null, client=10.40.198.13, traceId=068e76362589f1ee, method=PaginatedResponse com.oceanbase.ocp.server.common.controller.ComputeController.listHostTypes(Pageable), args=Page request [number: 0, size 5000, sort: id: DESC],] 2025-10-27 11:33:33.009 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,068e76362589f1ee,4a9ee3ff30c8d5ff] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/compute/hostTypes client=10.40.198.13, traceId=068e76362589f1ee, duration=9 ms] 2025-10-27 11:33:33.624 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:33.624 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:33.974 INFO 2458 --- [ocp-schedules-21,fe911d5cf6e1cf62,c42a15877164793a] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:33:33.974 INFO 2458 --- [ocp-schedules-21,fe911d5cf6e1cf62,c42a15877164793a] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:33:34.524 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:33:34.524 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:33:34.544 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:33:34.544 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:33:34.544 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:33:34.559 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:33:34.559 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:33:34.559 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:33:34.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,f340a415c3811be7,eef73fa28d946a36] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/time?null, client=10.40.198.13, traceId=f340a415c3811be7, method=OffsetDateTime com.oceanbase.ocp.server.common.controller.InfoController.time(), args=] 2025-10-27 11:33:34.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d16c33dd6b400dc3,d81bbce56f3c19a2] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/info?null, client=10.40.198.13, traceId=d16c33dd6b400dc3, method=OcpBuildInfo com.oceanbase.ocp.server.common.controller.InfoController.info(), args=] 2025-10-27 11:33:34.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,f340a415c3811be7,eef73fa28d946a36] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/time client=10.40.198.13, traceId=f340a415c3811be7, duration=10 ms] 2025-10-27 11:33:34.560 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,d16c33dd6b400dc3,d81bbce56f3c19a2] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/info client=10.40.198.13, traceId=d16c33dd6b400dc3, duration=10 ms] 2025-10-27 11:33:34.561 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,8335ffe5ac480ec3,d8031fc2ae69b15e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/sensitiveDataKey?null, client=10.40.198.13, traceId=8335ffe5ac480ec3, method=SuccessResponse com.oceanbase.ocp.server.common.controller.InfoController.getSensitiveDataPublicKey(), args=] 2025-10-27 11:33:34.561 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,8335ffe5ac480ec3,d8031fc2ae69b15e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/sensitiveDataKey client=10.40.198.13, traceId=8335ffe5ac480ec3, duration=11 ms] 2025-10-27 11:33:34.574 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:33:34.574 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:33:34.574 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:33:34.581 INFO 2458 --- [ocp-schedules-3,dc663eadf9b3ab99,c035447366451fce] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:33:34.583 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /error?null, client=10.40.198.13, traceId=, method=ModelAndView com.oceanbase.ocp.server.common.controller.web.CustomErrorController.handleError(HttpServletRequest), args=] 2025-10-27 11:33:34.583 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.c.web.CustomErrorController : Requested URL not found: [url=/api/v2/maz/state, method=10.40.198.13, client=GET, traceId=] 2025-10-27 11:33:34.583 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,,] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /error client=10.40.198.13, traceId=, duration=0 ms] 2025-10-27 11:33:34.642 INFO 2458 --- [ocp-async-6,66d8d7c66d376a32,4c565d17a4c96b57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:34.642 WARN 2458 --- [ocp-async-5,4836d1036b5da66e,a4e7c94cf2110eb2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:35.155 INFO 2458 --- [ocp-schedules-10,1c6beee596f0e122,c6488c9dbce7af4a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:35.178 WARN 2458 --- [scanner-112209226-1,1c6beee596f0e122,bfabd36cce7fd132] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:35.710 INFO 2458 --- [prometheus-minute-schedule-1,40082a477aa37f94,87cb2e4aa1f7fe26] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:35.711 INFO 2458 --- [ocp-schedules-4,3d41443cbeb7802e,71c6c7ec3feb3ecc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:35.715 INFO 2458 --- [ocp-schedules-4,3d41443cbeb7802e,71c6c7ec3feb3ecc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:35.716 INFO 2458 --- [prometheus-minute-schedule-1,40082a477aa37f94,87cb2e4aa1f7fe26] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:36.625 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:36.625 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:36.642 INFO 2458 --- [ocp-async-8,e12d48da75765bf5,c922d80c8455a9b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:36.642 WARN 2458 --- [ocp-async-2,376bc3b28cb2a68b,b8df46587668192e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:37.065 INFO 2458 --- [ocp-schedules-17,bffd0af3d16e5cbd,87080264844855c1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:37.521 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,b92a0b81b2b8efa4,707aee04994180c6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /api/v2/iam/actionGroups?null, client=10.40.198.13, traceId=b92a0b81b2b8efa4, method=IterableResponse com.oceanbase.ocp.server.common.controller.IamController.listActionGroups(), args=] 2025-10-27 11:33:37.527 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,b92a0b81b2b8efa4,707aee04994180c6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /api/v2/iam/actionGroups client=10.40.198.13, traceId=b92a0b81b2b8efa4, duration=13 ms] 2025-10-27 11:33:37.708 INFO 2458 --- [prometheus-cache-monitor1,62d92bda7b86c976,31bb380027c575f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20310,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5079,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:33:37.708 INFO 2458 --- [prometheus-cache-monitor1,62d92bda7b86c976,31bb380027c575f2] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:33:38.540 INFO 2458 --- [ocp-async-8,0cd5cdbb5cc23660,3e4fdd0600239872] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:38.642 INFO 2458 --- [ocp-async-1,1ab1bf08374c59a1,e502eb6a285d1694] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:38.642 WARN 2458 --- [ocp-async-10,10788aad0abcbca7,760b871a10397e37] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:38.707 INFO 2458 --- [prometheus-minute-schedule-1,17c8ce6316167c84,e557de52f5659f0d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:38.713 INFO 2458 --- [prometheus-minute-schedule-1,17c8ce6316167c84,e557de52f5659f0d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:38.834 INFO 2458 --- [ocp-schedules-5,2f810f8c3955490e,33fd40238d1490c6] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:33:38.841 INFO 2458 --- [ocp-schedules-5,2f810f8c3955490e,33fd40238d1490c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:38.841 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:33:38.842 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:33:38.842 INFO 2458 --- [ocp-schedules-5,2f810f8c3955490e,33fd40238d1490c6] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:33:39.625 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:39.626 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:39.939 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:39.939 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:33:39.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:33:39.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:33:39.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:33:39.940 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:33:39.951 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:39.952 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:39.953 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:39.953 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:39.953 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:39.955 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:39.956 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:39.956 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:39.957 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:39.957 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:40.007 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:33:40.007 INFO 2458 --- [ocp-schedules-28,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:33:40.010 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:33:40.011 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:33:40.025 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:33:40.025 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:33:40.027 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:33:40.029 INFO 2458 --- [ocp-schedules-21,8137677a5a15761c,20afba9018ef261f] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:33:40.070 INFO 2458 --- [Collect obproxy1,3cb53706b802e0c2,4c322750064172cc] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:33:40.070 INFO 2458 --- [Collect obproxy1,3cb53706b802e0c2,4c322750064172cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:33:40.070 INFO 2458 --- [Check obproxy1,b0f09d3a00271e36,f93d8a476af05470] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:33:40.070 INFO 2458 --- [Check obproxy1,b0f09d3a00271e36,f93d8a476af05470] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:33:40.100 INFO 2458 --- [Collect obproxy1,3cb53706b802e0c2,4c322750064172cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:33:40.100 INFO 2458 --- [Check obproxy1,b0f09d3a00271e36,f93d8a476af05470] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:33:40.103 INFO 2458 --- [Collect obproxy1,3cb53706b802e0c2,4c322750064172cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:33:40.103 INFO 2458 --- [Check obproxy1,b0f09d3a00271e36,f93d8a476af05470] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:33:40.103 INFO 2458 --- [Check obproxy1,b0f09d3a00271e36,f93d8a476af05470] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:33:40.106 INFO 2458 --- [Collect obproxy1,3cb53706b802e0c2,4c322750064172cc] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:33:40.189 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:33:40.189 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:33:40.283 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:40.285 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:33:40.296 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:40.297 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:33:40.356 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:40.358 INFO 2458 --- [ocp-schedules-1,7d0ddcfbf6fae8ca,3cd866aabb6df554] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:33:40.643 INFO 2458 --- [ocp-async-9,486f3590865f686f,8fbbfac0461590c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:40.643 WARN 2458 --- [ocp-async-6,3aa84e1331025d81,130e38b4911ac3e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:41.285 INFO 2458 --- [ocp-schedules-25,59c565371927f40d,647c126012817fa6] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:33:41.707 INFO 2458 --- [prometheus-minute-schedule-1,23bb13bb06e9f63b,1655082fa8a53ae1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:41.714 INFO 2458 --- [prometheus-minute-schedule-1,23bb13bb06e9f63b,1655082fa8a53ae1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:41.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:41.719631+08:00, lastProcessTime=2025-10-27T11:33:31.717293+08:00 2025-10-27 11:33:41.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:41.719631+08:00 2025-10-27 11:33:42.626 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:42.627 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:42.643 INFO 2458 --- [ocp-async-4,2e86bbaf15b05e73,9631f4ab39c56fcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:42.643 WARN 2458 --- [ocp-async-8,cf89ba58b028e9f0,21d5f0e556d0926b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:43.196 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,8088c1f54e1eef72,4573eb68a7ae24a0] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/software-packages, client 10.40.198.13, traceId 8088c1f54e1eef72] 2025-10-27 11:33:44.282 INFO 2458 --- [ocp-schedules-3,061ee1298e5db184,b0428b42649bc538] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:33:44.643 WARN 2458 --- [ocp-async-7,8737f15abfcbea70,afd9f8222a949717] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:44.643 INFO 2458 --- [ocp-async-10,1813be7fbe43f6dd,0a4c40d949adccba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:44.707 INFO 2458 --- [prometheus-minute-schedule-1,c7197290cda761c5,551914670b4dc3a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:44.715 INFO 2458 --- [prometheus-minute-schedule-1,c7197290cda761c5,551914670b4dc3a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:45.102 INFO 2458 --- [prometheus-second-schedule-1,91b456eb87493103,2bb3ab88c535232c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:33:45.286 INFO 2458 --- [ocp-schedules-16,57ce386f06526705,66de062e6a22c2bb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:45.307 WARN 2458 --- [scanner-112209388-1,57ce386f06526705,fd7b5137d4e26567] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:45.359 INFO 2458 --- [alarm-detect-11,57ce386f06526705,ff8b6349962e8e68] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761535955 ~ 1761536015 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535980, value=204314.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204314.56, values={1761535980=204314.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:33:45.360 INFO 2458 --- [alarm-detect-11,57ce386f06526705,ff8b6349962e8e68] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_compaction, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:33:45.360 INFO 2458 --- [alarm-detect-11,57ce386f06526705,ff8b6349962e8e68] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:33:45.449 INFO 2458 --- [alarm-detect-14,57ce386f06526705,8150f17bfca21314] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761535955 ~ 1761536015 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535980, value=204315.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204315.912, values={1761535980=204315.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:33:45.450 INFO 2458 --- [alarm-detect-14,57ce386f06526705,8150f17bfca21314] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_no_frozen, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:33:45.450 INFO 2458 --- [alarm-detect-14,57ce386f06526705,8150f17bfca21314] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:33:45.450 INFO 2458 --- [alarm-detect-17,57ce386f06526705,c89f79d6aa5d37fe] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761535955 ~ 1761536015 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761535980, value=93.214)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.214, values={1761535980=93.214}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:33:45.451 INFO 2458 --- [alarm-detect-17,57ce386f06526705,c89f79d6aa5d37fe] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_log_disk_usage_high, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2, currentState=Firing 2025-10-27 11:33:45.451 INFO 2458 --- [alarm-detect-17,57ce386f06526705,c89f79d6aa5d37fe] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:33:45.469 INFO 2458 --- [alarm-detect-13,57ce386f06526705,df542136c84dbab3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761535995 ~ 1761536015 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536010, value=59.612)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536010, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536010=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:33:45.627 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:45.628 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:45.727 INFO 2458 --- [ocp-schedules-4,e78ee38a9b0ddbb9,644f10adf4808299] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:45.730 INFO 2458 --- [ocp-schedules-4,e78ee38a9b0ddbb9,644f10adf4808299] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:46.643 INFO 2458 --- [ocp-async-5,9919a4727456752f,703a24fa76687de9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:46.643 WARN 2458 --- [ocp-async-6,96f6779504b88912,8602c6370da95608] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:47.072 INFO 2458 --- [ocp-schedules-30,9c427c3bbda38dbb,de398081e950f64c] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:47.707 INFO 2458 --- [prometheus-minute-schedule-1,1253fa3c5c8ede24,cde390fcfb46ac2a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:47.709 INFO 2458 --- [ocp-schedules-11,b4e28572c3a4585c,b111f97ac7cb3030] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:33:47.714 INFO 2458 --- [prometheus-minute-schedule-1,1253fa3c5c8ede24,cde390fcfb46ac2a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:47.726 INFO 2458 --- [ocp-schedules-11,b4e28572c3a4585c,b111f97ac7cb3030] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:47.726 INFO 2458 --- [ocp-schedules-11,b4e28572c3a4585c,b111f97ac7cb3030] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:33:48.541 INFO 2458 --- [ocp-async-5,e0c1b592f0aa2eae,e894935dfd24f65a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:48.628 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:48.628 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:48.643 INFO 2458 --- [ocp-async-2,c9c4302b18f13a50,e0453e5467f8f91c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:48.643 WARN 2458 --- [ocp-async-4,40928fbd867baea5,ffd69dd451a0fdbc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:49.875 INFO 2458 --- [ocp-schedules-26,ed1bdba554e54471,94c4a11d945edd64] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:33:49.875+08:00, clusters=[1, 2] 2025-10-27 11:33:49.890 INFO 2458 --- [ocp-schedules-26,ed1bdba554e54471,94c4a11d945edd64] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:33:49.892 INFO 2458 --- [ocp-schedules-26,ed1bdba554e54471,94c4a11d945edd64] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:33:50.643 WARN 2458 --- [ocp-async-10,501618ebc7d1a461,9120c2656234617b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:50.643 INFO 2458 --- [ocp-async-3,79d2cfb71e79c63c,d8ce21c33a0c992e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:50.707 INFO 2458 --- [prometheus-minute-schedule-1,9e8762b095b249cb,839e2843f240b84b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:50.714 INFO 2458 --- [prometheus-minute-schedule-1,9e8762b095b249cb,839e2843f240b84b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:51.184 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,f1437b7f69f6f2e4,158e15d21b51bc49] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=f1437b7f69f6f2e4, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:33:51.184 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,f1437b7f69f6f2e4,158e15d21b51bc49] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=f1437b7f69f6f2e4, duration=1 ms] 2025-10-27 11:33:51.409 INFO 2458 --- [ocp-schedules-1,f554aa3c4c773e8e,2e160e55bc14a1ec] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:51.409 INFO 2458 --- [ocp-schedules-30,d0b9eab2a247a515,d9aa6176cc45f015] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:51.409 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:33:51.409 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:33:51.410 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:33:51.410 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:33:51.410 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:33:51.411 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:33:51.411 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:33:51.411 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:33:51.411 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:33:51.411 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:33:51.424 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:51.425 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:51.425 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:51.428 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:51.428 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:51.428 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:51.428 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:51.429 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:51.430 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:51.430 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:51.431 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:51.431 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:51.432 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:51.432 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:51.433 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:51.433 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:51.433 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:51.434 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:51.435 INFO 2458 --- [ocp-schedules-19,805a8b54cacebf18,3e3b5c3a47674a89] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:33:51.435 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:51.435 INFO 2458 --- [host-check-clockdiff1,805a8b54cacebf18,a2c0690bcd1835be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:33:51.435 INFO 2458 --- [host-check-clockdiff2,805a8b54cacebf18,c4e8f6f9503c3c98] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:33:51.436 INFO 2458 --- [host-check-clockdiff3,805a8b54cacebf18,e9ee6c3ebf8e9826] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:33:51.436 INFO 2458 --- [host-check-clockdiff4,805a8b54cacebf18,77b1c3ece4dd5b2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:33:51.437 INFO 2458 --- [host-check-clockdiff5,805a8b54cacebf18,c84ef53c6fc79e6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:33:51.437 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:51.451 INFO 2458 --- [ocp-schedules-12,011e161aa7c686b1,b4be85d768594dc5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:33:51.474 INFO 2458 --- [host-check-clockdiff2,805a8b54cacebf18,c4e8f6f9503c3c98] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536031 -1 -1 , error= 2025-10-27 11:33:51.477 INFO 2458 --- [host-check-clockdiff4,805a8b54cacebf18,77b1c3ece4dd5b2e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536031 -1 -1 , error= 2025-10-27 11:33:51.483 INFO 2458 --- [host-check-clockdiff5,805a8b54cacebf18,c84ef53c6fc79e6a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536031 0 0 , error= 2025-10-27 11:33:51.483 INFO 2458 --- [host-check-clockdiff1,805a8b54cacebf18,a2c0690bcd1835be] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536031 0 0 , error= 2025-10-27 11:33:51.524 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:33:51.524 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:33:51.526 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:33:51.527 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:33:51.528 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:33:51.528302454+08:00 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status1,f554aa3c4c773e8e,f107e482c4a63dce] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status3,f554aa3c4c773e8e,ccd862332c11e04a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:33:51.528+08:00 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:33:51.529 INFO 2458 --- [host-check-agent-status4,f554aa3c4c773e8e,2367d7daf73ed604] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:33:51.530 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:33:51.531 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:33:51.531 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:33:51.531 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:33:51.530572121+08:00 2025-10-27 11:33:51.532 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:33:51.531+08:00 2025-10-27 11:33:51.532 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:33:51.531675144+08:00 2025-10-27 11:33:51.532 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:33:51.533 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:33:51.532+08:00 2025-10-27 11:33:51.533 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:33:51.533 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:33:51.533135402+08:00 2025-10-27 11:33:51.534 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:33:51.533+08:00 2025-10-27 11:33:51.535 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:33:51.535 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:33:51.535 INFO 2458 --- [host-check-agent-status5,f554aa3c4c773e8e,b7c8690424cb9cae] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:33:51.536 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:33:51.536 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:33:51.536 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:33:51.536 INFO 2458 --- [host-check-agent-status2,f554aa3c4c773e8e,a85de1fd26c5852f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:33:51.536 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:33:51.536811627+08:00 2025-10-27 11:33:51.537 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:33:51.536+08:00 2025-10-27 11:33:51.538 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:33:51.528+08:00 and status AVAILABLE 2025-10-27 11:33:51.540 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:33:51.532+08:00 and status AVAILABLE 2025-10-27 11:33:51.540 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:33:51.531+08:00 and status AVAILABLE 2025-10-27 11:33:51.541 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:33:51.533+08:00 and status AVAILABLE 2025-10-27 11:33:51.542 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:33:51.545 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:33:51.536+08:00 and status AVAILABLE 2025-10-27 11:33:51.545 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:33:51.545 INFO 2458 --- [ocp-schedules-3,79f888c754c99702,46d1bdc6fdbf193d] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:33:51.547 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:33:51.548 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:33:51.549 INFO 2458 --- [ocp-schedules-18,6dbf77e66e183d35,712de7d3ffd3ee7c] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:33:51.549 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:33:51.552 INFO 2458 --- [ocp-schedules-3,79f888c754c99702,46d1bdc6fdbf193d] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:51.553 INFO 2458 --- [ocp-schedules-18,6dbf77e66e183d35,712de7d3ffd3ee7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:51.554 INFO 2458 --- [ocp-schedules-11,439fd4b562942663,20871bdff91b6d13] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:33:51.554 INFO 2458 --- [ocp-schedules-18,6dbf77e66e183d35,712de7d3ffd3ee7c] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:33:51.559 INFO 2458 --- [ocp-schedules-11,439fd4b562942663,20871bdff91b6d13] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:51.565 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:33:51.566 INFO 2458 --- [ocp-schedules-27,1b145e6e949a8a62,a5f46ae344841b7e] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:33:51.568 INFO 2458 --- [ocp-schedules-17,97010283a41790af,480c196bd680476f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:33:51.569 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:33:51.569 INFO 2458 --- [ocp-schedules-6,3c159f55dac20b32,dc11210a52db5487] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:33:51.571 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:33:51.573 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:33:51.573 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:51.574 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:33:51.574 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:51.576 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:51.576 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:33:51.577 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:51.580 INFO 2458 --- [ocp-schedules-17,97010283a41790af,480c196bd680476f] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:33:51.581 INFO 2458 --- [ocp-schedules-6,3c159f55dac20b32,dc11210a52db5487] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:33:51.606 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.606 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:51.607 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:33:51.618 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.619 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:51.629 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:51.629 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:33:51.636 INFO 2458 --- [ocp-schedules-3,79f888c754c99702,46d1bdc6fdbf193d] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:33:51.638 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:51.639 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:51.641 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:51.642 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:51.644 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:33:51.654 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:33:51.656 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:33:51.662 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:33:51.666 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:33:51.673 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:33:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:51.686 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:33:51.689 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:33:51.691 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:51.692 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.693 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:33:51.694 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:33:51.700 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:33:51.700 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:33:51.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:33:51.715210+08:00, lastProcessTime=2025-10-27T11:33:41.719631+08:00 2025-10-27 11:33:51.719 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:51.721 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:51.727 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:51.734 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:33:51.734 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:51.738 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:33:51.739 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:33:51.748 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:33:51.753 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:51.754 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.754 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:51.755 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.758 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:33:51.759 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:51.761 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:33:51.768 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:33:51.776 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:33:51.776 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:33:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:51.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:51.783 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:33:51.788 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:51.789 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.789 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:33:51.805 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:33:51.808 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:33:51.809 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:51.809 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:51.815 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:51.818 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:51.827 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:33:51.830 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:33:51.839 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:51.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:51.839 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.840 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:33:51.840 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:51.841 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:33:51.852 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:33:51.859 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:33:51.872 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:33:51.873 INFO 2458 --- [host-check-host-status4,d0b9eab2a247a515,a172687b4ba12e2b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:33:51.876 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:33:51.877 INFO 2458 --- [host-check-host-status5,d0b9eab2a247a515,ca52ae62c7a5215b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:33:51.878 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:33:51.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:33:51.891 INFO 2458 --- [host-check-host-status2,d0b9eab2a247a515,eba19356dcf1bee7] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:33:51.897 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:51.899 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.899 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:33:51.901 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.903 INFO 2458 --- [ocp-schedules-23,4967454d5b49707b,03f8f1baca8fabf0] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:33:51.904 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:33:51.905 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:51.912 INFO 2458 --- [host-check-host-status1,d0b9eab2a247a515,b368204ba9162159] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:33:51.916 INFO 2458 --- [host-check-host-status3,d0b9eab2a247a515,1ea5ef0ae6e1b8b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:33:51.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:33:51.947 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:51.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:51.986 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:33:51.986 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.987 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:33:51.990 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:51.991 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:33:52.008 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:33:51.715210+08:00 2025-10-27 11:33:52.008 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:52.009 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:52.010 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:33:52.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:52.034 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:52.035 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:33:52.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:52.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:33:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:33:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:52.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:33:52.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:52.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:33:52.157 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:33:52.197 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:52.198 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:52.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:33:52.202 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:33:52.212 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:33:52.222 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:33:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:33:52.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:33:52.337 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:33:52.337 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:52.338 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:33:52.340 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:52.341 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:33:52.375 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:33:52.383 INFO 2458 --- [host-check-clockdiff3,805a8b54cacebf18,e9ee6c3ebf8e9826] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536032 0 0 , error= 2025-10-27 11:33:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:33:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:33:52.417 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:33:52.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:33:52.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:33:52.463 INFO 2458 --- [ocp-schedules-11,439fd4b562942663,20871bdff91b6d13] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:33:52.643 WARN 2458 --- [ocp-async-9,b1b4228e8ea4ab63,99184bb53e45552b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:52.643 INFO 2458 --- [ocp-async-9,5de9665ee7635732,35afb880697d2321] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:33:53.236 INFO 2458 --- [ocp-schedules-30,66ea6dff291dc0c9,6a697cde16d4e1bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20310,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5079,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:33:53.236 INFO 2458 --- [ocp-schedules-30,66ea6dff291dc0c9,6a697cde16d4e1bb] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:33:53.707 INFO 2458 --- [prometheus-minute-schedule-1,f04923335664900a,a0eeb8a7b75d1376] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:53.714 INFO 2458 --- [prometheus-minute-schedule-1,f04923335664900a,a0eeb8a7b75d1376] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:54.629 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:54.630 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:54.644 INFO 2458 --- [ocp-async-8,e5f7bec86d3ca770,7b1a2cf8b9579d65] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:54.644 WARN 2458 --- [ocp-async-2,29f35a43d005e8a1,cfffbc963f02ac33] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:55.496 INFO 2458 --- [ocp-schedules-23,f8dc6eaec605dc10,c445d2de255ff1a1] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:33:55.514 WARN 2458 --- [scanner-112209649-1,f8dc6eaec605dc10,dd32d620f4fc4c1f] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:33:55.626 INFO 2458 --- [alarm-detect-17,f8dc6eaec605dc10,be118286e9680bd1] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536010 ~ 1761536025 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536025, value=58.621)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536025, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536025=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:33:55.662 INFO 2458 --- [ocp-schedules-25,2040ac2328b9c0ed,a377d67e971102e5] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:33:55.737 INFO 2458 --- [ocp-schedules-3,3ce7c76c9f5f6731,5f5000bb9ea074d8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:33:55.741 INFO 2458 --- [ocp-schedules-3,3ce7c76c9f5f6731,5f5000bb9ea074d8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:33:56.644 INFO 2458 --- [ocp-async-7,d3989e8a1fa3710b,4641e17dfd7cf480] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:56.644 WARN 2458 --- [ocp-async-6,dec25f13f06f0aec,39ba8353c91be033] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:56.707 INFO 2458 --- [prometheus-minute-schedule-1,ffedf7dcf94b511e,6fe0a2bfa53a340b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:56.714 INFO 2458 --- [prometheus-minute-schedule-1,ffedf7dcf94b511e,6fe0a2bfa53a340b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:33:57.078 INFO 2458 --- [ocp-schedules-14,8f0e774e23907b8d,13ff4030d097c17d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:33:57.630 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:33:57.631 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:33:58.400 INFO 2458 --- [ocp-schedules-15,ad8933df3cfa32fb,3bbb3ba6125c62b6] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:33:58.407 INFO 2458 --- [ocp-schedules-15,ad8933df3cfa32fb,3bbb3ba6125c62b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:33:58.453 INFO 2458 --- [ocp-schedules-15,ad8933df3cfa32fb,3bbb3ba6125c62b6] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:33:58.541 INFO 2458 --- [ocp-async-8,6e73a40a55e5279f,e552f868cc8f250f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:33:58.644 INFO 2458 --- [ocp-async-4,35da7a951b114bae,f4d6ea0df8155d9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:33:58.644 WARN 2458 --- [ocp-async-5,9dcd18adfd76d86f,d0fc3e0af70e2f85] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:33:59.550 INFO 2458 --- [ocp-schedules-18,449fcf679de04ebd,78deae9be2bcc600] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:33:59.622 INFO 2458 --- [ocp-schedules-13,c22aac9933487fd5,68439e356b020978] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:33:59.624 INFO 2458 --- [ocp-schedules-13,c22aac9933487fd5,68439e356b020978] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:33:59.707 INFO 2458 --- [prometheus-minute-schedule-1,501b081163a8407e,6ce407ad55a625bd] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536040, interval=60 2025-10-27 11:33:59.713 INFO 2458 --- [prometheus-minute-schedule-1,501b081163a8407e,6ce407ad55a625bd] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:00.631 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:00.632 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:00.644 INFO 2458 --- [ocp-async-4,0f8fa125c0a6bf9d,76fb1b92382b3522] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:00.644 WARN 2458 --- [ocp-async-10,564fdfcaf64fafde,35432a6a917a52eb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:01.714934+08:00, lastProcessTime=2025-10-27T11:33:51.715210+08:00 2025-10-27 11:34:01.940 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:01.714934+08:00 2025-10-27 11:34:02.644 WARN 2458 --- [ocp-async-3,15b295da395eff47,6303b44b55150cd7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:02.644 INFO 2458 --- [ocp-async-4,06872338e3d73b14,277bacef022803cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:02.707 INFO 2458 --- [prometheus-minute-schedule-1,80ebee5e389ad4b8,1b8185dfcacbc483] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:02.714 INFO 2458 --- [prometheus-minute-schedule-1,80ebee5e389ad4b8,1b8185dfcacbc483] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:03.632 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:03.633 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:04.644 INFO 2458 --- [ocp-async-5,4c1fe219a2358caf,0e3e260a077bb066] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:04.644 WARN 2458 --- [ocp-async-2,ebb111150a6428e5,d50b28916108b4a2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:04.763 INFO 2458 --- [ocp-schedules-6,3ddf0b8824f73240,383dc396f6c7a538] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:34:04.772 INFO 2458 --- [ocp-schedules-6,3ddf0b8824f73240,383dc396f6c7a538] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:04.807 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:04.808 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:34:04.824 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:04.825 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:34:04.838 INFO 2458 --- [ocp-schedules-6,3ddf0b8824f73240,383dc396f6c7a538] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:34:05.631 INFO 2458 --- [ocp-schedules-13,ca2884cb6b59f5b2,d8a8ca388f9cd501] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:05.690 WARN 2458 --- [scanner-112209855-1,ca2884cb6b59f5b2,a4722f5a37527d61] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:05.707 INFO 2458 --- [prometheus-minute-schedule-1,7867f292ba63afc5,74fb2c7c4c503242] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:05.724 INFO 2458 --- [prometheus-minute-schedule-1,7867f292ba63afc5,74fb2c7c4c503242] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:05.748 INFO 2458 --- [ocp-schedules-26,7d692129cb2c2685,1aa71d658dc2dbb3] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:05.752 INFO 2458 --- [ocp-schedules-26,7d692129cb2c2685,1aa71d658dc2dbb3] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:34:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:34:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:34:06.078 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:34:06.118 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:34:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:34:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:34:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:34:06.143 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:34:06.145 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:34:06.145 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:34:06.145 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:34:06.145 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:34:06.148 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:34:06.149 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:34:06.149 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:06.150 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:34:06.152 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:06.152 INFO 2458 --- [sql-rollup-worker-3,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:34:06.226 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:31:30, start=2025-10-27T11:27:30, end=2025-10-27T11:30:00, updates=234, granule=120000ms, delay=240000ms 2025-10-27 11:34:06.234 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:31:30, target=2025-10-27T11:30:00, granule=120000ms, delay=90000ms) 2025-10-27 11:34:06.234 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:30:00, target=2025-10-27T11:22:30, granule=600000ms, delay=450000ms) 2025-10-27 11:34:06.234 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=230ms, period=120000ms 2025-10-27 11:34:06.633 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:06.634 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:06.644 WARN 2458 --- [ocp-async-5,785ecde1142277c2,941e4eccb1099c6d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:06.645 INFO 2458 --- [ocp-async-5,3c6571e7ec575e7d,140fcfe30dc44171] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:07.086 INFO 2458 --- [ocp-schedules-5,26c65998cf64a304,45c7f7364cdc888f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:07.729 INFO 2458 --- [ocp-schedules-7,2ef2e1b26b99ea2c,16455cb10c0d64c7] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:34:07.737 INFO 2458 --- [ocp-schedules-7,2ef2e1b26b99ea2c,16455cb10c0d64c7] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:07.737 INFO 2458 --- [ocp-schedules-7,2ef2e1b26b99ea2c,16455cb10c0d64c7] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:08.064 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bc675734ad76808e,16c973da2484be6b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=bc675734ad76808e, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:34:08.065 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-6,bc675734ad76808e,16c973da2484be6b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=bc675734ad76808e, duration=1 ms] 2025-10-27 11:34:08.541 INFO 2458 --- [ocp-async-7,9d6968605fb07258,569eda86af8475ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:08.645 INFO 2458 --- [ocp-async-3,848dd316419bf9ce,0f95d7836d625962] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:08.645 WARN 2458 --- [ocp-async-4,2c3e02520647ec7c,89ce4ca598ef3a11] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:08.707 INFO 2458 --- [prometheus-minute-schedule-1,6a89b814b1242541,4e0bee0881c01d66] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:08.714 INFO 2458 --- [prometheus-minute-schedule-1,6a89b814b1242541,4e0bee0881c01d66] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:09.634 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:09.634 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:10.009 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:34:10.010 INFO 2458 --- [ocp-schedules-22,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:34:10.043 INFO 2458 --- [Check obproxy1,8244b3eb086e8858,5e32e761e00a74a1] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:34:10.044 INFO 2458 --- [Check obproxy1,8244b3eb086e8858,5e32e761e00a74a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:34:10.044 INFO 2458 --- [Collect obproxy1,b00bbdba333e4c31,a410655ed9f3aef5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:34:10.044 INFO 2458 --- [Collect obproxy1,b00bbdba333e4c31,a410655ed9f3aef5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:34:10.072 INFO 2458 --- [Collect obproxy1,b00bbdba333e4c31,a410655ed9f3aef5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:34:10.075 INFO 2458 --- [Check obproxy1,8244b3eb086e8858,5e32e761e00a74a1] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:34:10.076 INFO 2458 --- [Collect obproxy1,b00bbdba333e4c31,a410655ed9f3aef5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:34:10.079 INFO 2458 --- [Check obproxy1,8244b3eb086e8858,5e32e761e00a74a1] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:34:10.079 INFO 2458 --- [Check obproxy1,8244b3eb086e8858,5e32e761e00a74a1] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:34:10.080 INFO 2458 --- [Collect obproxy1,b00bbdba333e4c31,a410655ed9f3aef5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:34:10.183 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:34:10.484 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=1000, affectRows=1000 2025-10-27 11:34:10.488 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=994, validCount=992 2025-10-27 11:34:10.506 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505683) Duplicate entry '835370602979598336-1761535920' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505683) Duplicate entry '835370602979598336-1761535920' for key 'PRIMARY' 2025-10-27 11:34:10.560 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=1000, affectRows=1000 2025-10-27 11:34:10.564 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=999, validCount=995 2025-10-27 11:34:10.613 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=1000, affectRows=1000 2025-10-27 11:34:10.618 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=996, validCount=992 2025-10-27 11:34:10.662 INFO 2458 --- [ocp-async-3,3c0082eb81021743,b2d22b7b91683373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:10.662 WARN 2458 --- [ocp-async-10,823eba169c35670e,2e13dda5e72129d8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:10.663 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=1000, affectRows=1000 2025-10-27 11:34:10.667 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=996, validCount=992 2025-10-27 11:34:10.682 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=79, affectRows=79 2025-10-27 11:34:10.682 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=78, validCount=78 2025-10-27 11:34:11.707 INFO 2458 --- [prometheus-minute-schedule-1,373c470fb60ee3de,eb3e197497f731f8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:11.712787+08:00, lastProcessTime=2025-10-27T11:34:01.714934+08:00 2025-10-27 11:34:11.714 INFO 2458 --- [prometheus-minute-schedule-1,373c470fb60ee3de,eb3e197497f731f8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:11.932 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:11.712787+08:00 2025-10-27 11:34:12.635 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:12.635 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:12.662 WARN 2458 --- [ocp-async-3,290d0c83a867b789,b10760538ddd4e09] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:12.662 INFO 2458 --- [ocp-async-6,2c990ba12b5168e7,50eda5a235eabd3d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:12.668 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761535980, nodeCount=817, affectRows=817 2025-10-27 11:34:14.662 INFO 2458 --- [ocp-async-6,79c3cbbce81507b2,56da0bdb7d5e5c58] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:14.662 WARN 2458 --- [ocp-async-2,b56d4310b7093d11,845b9a6a407b9268] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:14.707 INFO 2458 --- [prometheus-minute-schedule-1,819c32bad9c1e94f,9d254eb01e6f5b73] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:14.714 INFO 2458 --- [prometheus-minute-schedule-1,819c32bad9c1e94f,9d254eb01e6f5b73] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:14.764 INFO 2458 --- [ocp-schedules-30,abb09c60dc4a39cb,671398a79b3c0ca0] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:34:14.771 INFO 2458 --- [ocp-schedules-30,abb09c60dc4a39cb,671398a79b3c0ca0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:14.856 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:34:14.857 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.860 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:34:14.869 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.870 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:14.897 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.898 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:14.922 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:34:14.922 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.925 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:34:14.926 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.927 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:14.929 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.929 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:14.949 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.950 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:14.962 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.963 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:14.978 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.979 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:14.991 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:14.992 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:15.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.009 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:34:15.011 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.012 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:15.018 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761535980, nodeCount=1000, affectRows=1000 2025-10-27 11:34:15.022 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761535980, nodeCount=1000, validCount=987 2025-10-27 11:34:15.028 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.029 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.029 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.030 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.031 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.032 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.033 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.034 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.035 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.036 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.037 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.037 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:34:15.040 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:34:15.041 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.042 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:15.071 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.072 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:34:15.075 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.076 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:34:15.093 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:15.094 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.095 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:34:15.104 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.104 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:34:15.104 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.104 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:34:15.108 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:34:15.108 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:34:15.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.110 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:34:15.144 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.144 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:34:15.147 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.148 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:34:15.179 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.179 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:34:15.210 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.210 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:34:15.258 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.259 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.260 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.261 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.262 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.263 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.264 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.265 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.265 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.266 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.267 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.268 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.269 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.269 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.270 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.270 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.271 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:34:15.272 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.272 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:34:15.273 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.274 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:34:15.274 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:34:15.277 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:34:15.365 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:15.366 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:15.367 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:34:15.391 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.391 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:34:15.392 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:34:15.396 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:34:15.396 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:34:15.396 INFO 2458 --- [ocp-schedules-30,abb09c60dc4a39cb,671398a79b3c0ca0] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:34:15.635 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:15.636 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:15.762 INFO 2458 --- [ocp-schedules-27,85f62ada2ba08ec5,41190424d75d2bfb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:15.766 INFO 2458 --- [ocp-schedules-27,85f62ada2ba08ec5,41190424d75d2bfb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:15.834 INFO 2458 --- [ocp-schedules-6,d554b64511a030bc,c8c88c15799bc5a3] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:15.889 WARN 2458 --- [scanner-112211653-1,d554b64511a030bc,1d601ef5ccb51644] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:16.201 INFO 2458 --- [alarm-detect-1,d554b64511a030bc,626886163dfc7598] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536025 ~ 1761536046 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536040, value=61.06)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536040, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536040=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:34:16.301 INFO 2458 --- [ocp-schedules-19,52110d4aa6d9a6b3,ba7f7527c0aa5a36] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:34:16.301+08:00, clusters=[1, 2] 2025-10-27 11:34:16.304 INFO 2458 --- [ocp-schedules-19,52110d4aa6d9a6b3,ba7f7527c0aa5a36] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:34:16.305 INFO 2458 --- [ocp-schedules-19,52110d4aa6d9a6b3,ba7f7527c0aa5a36] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:34:16.662 INFO 2458 --- [ocp-async-10,45b18ec42dc04cde,41a2ea03a384fe1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:16.662 WARN 2458 --- [ocp-async-1,e5851626b72e32e9,ab9a03ab69207e61] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:17.092 INFO 2458 --- [ocp-schedules-10,227d2d5b6dd129a4,db6a2cef8e984119] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:17.707 INFO 2458 --- [prometheus-minute-schedule-1,bae35d3552703926,f1df3eda46133c43] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:17.714 INFO 2458 --- [prometheus-minute-schedule-1,bae35d3552703926,f1df3eda46133c43] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:18.205 INFO 2458 --- [ocp-schedules-29,4c744f3b2c65f049,6fa8c80c55cdc2b7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:34:18.205 INFO 2458 --- [ocp-schedules-29,4c744f3b2c65f049,6fa8c80c55cdc2b7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:34:18.255 INFO 2458 --- [Collect all tenants major compactions2,4c744f3b2c65f049,6e01ccd07061e13f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:34:18.255 INFO 2458 --- [Collect all tenants major compactions1,4c744f3b2c65f049,cc37dca8e85951fb] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:34:18.255 INFO 2458 --- [Collect all tenants major compactions5,4c744f3b2c65f049,fc53caa657b695a7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:34:18.256 INFO 2458 --- [Collect all tenants major compactions3,4c744f3b2c65f049,25c94caf6df891e5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:34:18.256 INFO 2458 --- [Collect all tenants major compactions4,4c744f3b2c65f049,dcce6266e6db02c9] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:34:18.256 INFO 2458 --- [Collect all tenants major compactions6,4c744f3b2c65f049,197bb8477b063ef2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:34:18.292 INFO 2458 --- [Collect all tenants major compactions6,4c744f3b2c65f049,197bb8477b063ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.293 INFO 2458 --- [Collect all tenants major compactions5,4c744f3b2c65f049,fc53caa657b695a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.294 INFO 2458 --- [Collect all tenants major compactions6,4c744f3b2c65f049,197bb8477b063ef2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:34:18.295 INFO 2458 --- [Collect all tenants major compactions5,4c744f3b2c65f049,fc53caa657b695a7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:34:18.297 INFO 2458 --- [Collect all tenants major compactions6,4c744f3b2c65f049,197bb8477b063ef2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:34:18.303 INFO 2458 --- [Collect all tenants major compactions5,4c744f3b2c65f049,fc53caa657b695a7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:34:18.315 INFO 2458 --- [Collect all tenants major compactions3,4c744f3b2c65f049,25c94caf6df891e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.315 INFO 2458 --- [Collect all tenants major compactions4,4c744f3b2c65f049,dcce6266e6db02c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.316 INFO 2458 --- [Collect all tenants major compactions3,4c744f3b2c65f049,25c94caf6df891e5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:34:18.317 INFO 2458 --- [Collect all tenants major compactions4,4c744f3b2c65f049,dcce6266e6db02c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:34:18.320 INFO 2458 --- [Collect all tenants major compactions3,4c744f3b2c65f049,25c94caf6df891e5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:34:18.320 INFO 2458 --- [Collect all tenants major compactions4,4c744f3b2c65f049,dcce6266e6db02c9] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:34:18.327 INFO 2458 --- [Collect all tenants major compactions1,4c744f3b2c65f049,cc37dca8e85951fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.327 INFO 2458 --- [Collect all tenants major compactions2,4c744f3b2c65f049,6e01ccd07061e13f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:18.328 INFO 2458 --- [Collect all tenants major compactions2,4c744f3b2c65f049,6e01ccd07061e13f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:34:18.328 INFO 2458 --- [Collect all tenants major compactions1,4c744f3b2c65f049,cc37dca8e85951fb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:34:18.331 INFO 2458 --- [Collect all tenants major compactions2,4c744f3b2c65f049,6e01ccd07061e13f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:34:18.331 INFO 2458 --- [Collect all tenants major compactions1,4c744f3b2c65f049,cc37dca8e85951fb] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:34:18.332 INFO 2458 --- [ocp-schedules-29,4c744f3b2c65f049,6fa8c80c55cdc2b7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:34:18.541 INFO 2458 --- [ocp-async-10,48ec84fd9e5a8a85,9649f3399bf82448] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:18.636 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:18.637 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:18.662 WARN 2458 --- [ocp-async-5,ff83f87f3971b290,a76d0e71a6a03be6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:18.662 INFO 2458 --- [ocp-async-7,3e08e471b8c43239,08136f660287ea8d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:19.028 INFO 2458 --- [ocp-schedules-7,914b029b8de52023,c96d3ad17bc1b5c4] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:19.028 INFO 2458 --- [ocp-schedules-7,914b029b8de52023,c96d3ad17bc1b5c4] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:34:20.663 INFO 2458 --- [ocp-async-2,62ba930f875a2133,027f8537367e1dcb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:20.663 WARN 2458 --- [ocp-async-8,4560894c000f9570,b005b03e421932c5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:20.692 INFO 2458 --- [ocp-schedules-26,9a9924f61af418aa,c3acaefccdbff4af] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:34:20.707 INFO 2458 --- [prometheus-minute-schedule-1,e7e025d2d607d925,e70662246cac4ddf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:20.713 INFO 2458 --- [prometheus-minute-schedule-1,e7e025d2d607d925,e70662246cac4ddf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:21.407 INFO 2458 --- [ocp-schedules-22,cfbd3104a16c2b27,84ecf15660830967] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:34:21.408 INFO 2458 --- [ocp-schedules-12,5d1bb3bbd0f691d9,14cca1f56d478d49] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:34:21.408 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:34:21.409 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:34:21.409 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:34:21.409 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:34:21.410 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:34:21.426 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:21.427 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:21.427 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:21.427 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:21.428 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:21.429 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:21.432 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:21.432 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:21.432 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:21.432 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:21.432 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:21.433 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:21.433 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:21.433 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:21.433 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:21.433 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:21.436 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:21.436 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:21.436 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:21.437 INFO 2458 --- [ocp-schedules-3,e104ae07a67c2d2d,eaddc7ebb78efad4] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:21.437 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:21.437 INFO 2458 --- [host-check-clockdiff1,e104ae07a67c2d2d,2aae7a824cacf909] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:34:21.437 INFO 2458 --- [host-check-clockdiff2,e104ae07a67c2d2d,0b34268faaa5700b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:34:21.438 INFO 2458 --- [host-check-clockdiff3,e104ae07a67c2d2d,efdc513f96e6e7a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:34:21.438 INFO 2458 --- [host-check-clockdiff4,e104ae07a67c2d2d,27dfe778835ea7d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:34:21.440 INFO 2458 --- [host-check-clockdiff5,e104ae07a67c2d2d,344603d2cb2f4427] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:34:21.446 INFO 2458 --- [ocp-schedules-2,71efa9bbf610fd6f,d6dd9943d5f5681c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:34:21.466 INFO 2458 --- [host-check-clockdiff4,e104ae07a67c2d2d,27dfe778835ea7d5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536061 0 0 , error= 2025-10-27 11:34:21.482 INFO 2458 --- [host-check-clockdiff3,e104ae07a67c2d2d,efdc513f96e6e7a8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536061 0 0 , error= 2025-10-27 11:34:21.485 INFO 2458 --- [host-check-clockdiff2,e104ae07a67c2d2d,0b34268faaa5700b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536061 -1 -1 , error= 2025-10-27 11:34:21.485 INFO 2458 --- [host-check-clockdiff5,e104ae07a67c2d2d,344603d2cb2f4427] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536061 0 0 , error= 2025-10-27 11:34:21.488 INFO 2458 --- [host-check-clockdiff1,e104ae07a67c2d2d,2aae7a824cacf909] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536061 0 0 , error= 2025-10-27 11:34:21.519 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:34:21.523 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:34:21.523 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:34:21.523 INFO 2458 --- [host-check-agent-status3,5d1bb3bbd0f691d9,b6c15d9d7af0524b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:34:21.524 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:34:21.524 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:34:21.524 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:34:21.524 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:34:21.525 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:34:21.524475948+08:00 2025-10-27 11:34:21.525 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:34:21.525 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:34:21.525 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:34:21.525+08:00 2025-10-27 11:34:21.525 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:34:21.525356658+08:00 2025-10-27 11:34:21.528 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:34:21.525+08:00 2025-10-27 11:34:21.528 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:34:21.527456847+08:00 2025-10-27 11:34:21.529 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:34:21.528870855+08:00 2025-10-27 11:34:21.529 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:34:21.529+08:00 2025-10-27 11:34:21.529 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:34:21.528+08:00 2025-10-27 11:34:21.530 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:34:21.530 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:34:21.530 INFO 2458 --- [host-check-agent-status4,5d1bb3bbd0f691d9,62df122782ab6a60] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:34:21.531 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:34:21.531 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:34:21.531 INFO 2458 --- [host-check-agent-status1,5d1bb3bbd0f691d9,64b53eb82c958c5e] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:34:21.536 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:34:21.525+08:00 and status AVAILABLE 2025-10-27 11:34:21.540 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:34:21.540 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:34:21.525+08:00 and status AVAILABLE 2025-10-27 11:34:21.541 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:34:21.541 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:34:21.528+08:00 and status AVAILABLE 2025-10-27 11:34:21.542 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:34:21.529+08:00 and status AVAILABLE 2025-10-27 11:34:21.542 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:34:21.544 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status5,5d1bb3bbd0f691d9,2695a0b536f27fbd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-agent-status2,5d1bb3bbd0f691d9,6383b202eef09e5c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:34:21.545 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:34:21.545895631+08:00 2025-10-27 11:34:21.545 INFO 2458 --- [ocp-schedules-19,6f90f977498ded3b,2ad10ea910ca4bf8] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:34:21.546 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:34:21.546 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:34:21.545+08:00 2025-10-27 11:34:21.548 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:34:21.548 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:34:21.549 INFO 2458 --- [ocp-schedules-6,0d4c4d7127770ef1,6b613d9866398053] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:34:21.549 INFO 2458 --- [ocp-schedules-19,6f90f977498ded3b,2ad10ea910ca4bf8] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:21.553 INFO 2458 --- [ocp-schedules-6,0d4c4d7127770ef1,6b613d9866398053] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:21.553 INFO 2458 --- [ocp-schedules-6,0d4c4d7127770ef1,6b613d9866398053] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:34:21.554 INFO 2458 --- [ocp-schedules-24,f11f5ad0c1d2c8ab,02f0e8e4f8084dcc] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:34:21.554 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:34:21.545+08:00 and status AVAILABLE 2025-10-27 11:34:21.558 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:34:21.559 INFO 2458 --- [ocp-schedules-24,f11f5ad0c1d2c8ab,02f0e8e4f8084dcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:21.564 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:21.566 INFO 2458 --- [ocp-schedules-29,d85518b1b8f94f22,7f9bd811d98ebe91] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:34:21.567 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:21.567 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:21.569 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:21.570 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:21.571 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:21.572 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:21.574 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:21.581 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:21.584 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:21.589 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.590 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:21.592 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:34:21.609 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.610 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:34:21.617 INFO 2458 --- [ocp-schedules-19,6f90f977498ded3b,2ad10ea910ca4bf8] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:34:21.627 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:21.630 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.630 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:21.634 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:21.634 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:21.634 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:21.637 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:21.638 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:21.644 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:34:21.646 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:21.653 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:34:21.654 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:34:21.665 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:34:21.666 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:34:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.679 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:21.683 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:34:21.692 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:34:21.696 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:34:21.698 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:34:21.701 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:34:21.708 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:34:21.716 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:21.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:21.715066+08:00, lastProcessTime=2025-10-27T11:34:11.712787+08:00 2025-10-27 11:34:21.727 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:21.731 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:34:21.744 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:21.745 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:21.749 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:34:21.750 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:21.765 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:34:21.767 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:34:21.772 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:34:21.776 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:34:21.780 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:21.781 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:21.781 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:34:21.782 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.786 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:34:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:21.792 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:34:21.813 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:34:21.813 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:34:21.814 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:21.818 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:21.822 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:34:21.828 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:34:21.833 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:21.847 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:21.853 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:21.853 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:21.854 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:34:21.859 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:21.860 INFO 2458 --- [host-check-host-status4,cfbd3104a16c2b27,32a4a972631685c0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:34:21.866 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:34:21.879 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:34:21.882 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:21.882 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:34:21.883 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.883 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:21.885 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:34:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:21.886 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:21.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:21.891 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:34:21.892 INFO 2458 --- [host-check-host-status3,cfbd3104a16c2b27,20d3770e59c82cf4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:34:21.892 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:21.896 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:34:21.899 INFO 2458 --- [host-check-host-status5,cfbd3104a16c2b27,8f367ee101d987fb] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:34:21.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:21.907 INFO 2458 --- [ocp-schedules-4,4de8f9f7e220f0ff,eecd382f8b1f2705] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:34:21.910 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:21.916 INFO 2458 --- [host-check-host-status2,cfbd3104a16c2b27,3db08880a295a081] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:34:21.918 INFO 2458 --- [host-check-host-status1,cfbd3104a16c2b27,233c7757a96f2c26] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:34:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:21.991 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:34:22.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:22.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:22.009 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:21.715066+08:00 2025-10-27 11:34:22.033 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:34:22.034 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.034 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:34:22.036 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:34:22.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:22.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:22.053 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:34:22.075 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:22.075 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.076 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:34:22.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:22.087 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:22.099 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:34:22.100 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:34:22.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:22.121 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:22.158 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:22.197 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:22.198 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:22.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:34:22.212 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:34:22.221 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:22.224 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:22.269 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:34:22.342 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:34:22.342 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.343 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:34:22.345 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.346 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:34:22.379 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:34:22.421 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:22.422 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:22.423 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:34:22.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:34:22.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:34:22.463 INFO 2458 --- [ocp-schedules-24,f11f5ad0c1d2c8ab,02f0e8e4f8084dcc] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:34:22.663 INFO 2458 --- [ocp-async-1,d9a814317302ebf4,e4c81db5fe87cfe4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:22.663 WARN 2458 --- [ocp-async-10,1b927a3809613812,03312a1eb3f7361a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:23.331 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325} 2025-10-27 11:34:23.340 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocp 2025-10-27 11:34:23.340 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:23.342 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:23.342 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:34:23.342 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:23.345 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:23.350 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:23.350 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:34:23.350 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:23.358 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:23.360 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:23.360 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:34:23.360 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:23.362 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:23.364 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:23.364 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:34:23.364 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:23.366 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:23.373 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:23.373 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:34:23.373 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:23.485 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449663485000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:34:23.485Z, endTime=2025-10-27T03:32:23.325Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:34:23.500 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449663500000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:34:23.500Z, endTime=2025-10-27T03:32:23.325Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:34:23.539 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2025-10-27 11:34:23.539 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:34:23.563 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:34:23.563 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:34:23.573 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:34:23.573 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:34:23.597 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:34:23.597 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:34:23.603 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:34:23.613 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:34:23.613 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:34:23.644 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:34:23.644 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:34:23.649 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:34:23.649 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:34:23.675 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:34:23.675 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761535943325000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:34:23.679 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:34:23.680 INFO 2458 --- [plan_change_detector4,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:34:23.485, end=2025-10-27T11:32:23.325}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:34:23.707 INFO 2458 --- [prometheus-minute-schedule-1,e8caa7b4a307fc9f,818127082ddbf1f4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:23.717 INFO 2458 --- [prometheus-minute-schedule-1,e8caa7b4a307fc9f,818127082ddbf1f4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:24.631 INFO 2458 --- [plan_change_detector7,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:34:23.5, end=2025-10-27T11:32:23.325}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:34:24.632 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:34:24.634 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:22.562, end=2025-10-27T11:32:23.325},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:34:24.634 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:34:24.634 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:34:24.636 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:34:24.638 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:24.639 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:24.663 WARN 2458 --- [ocp-async-3,1f5b2fd518266182,87706f8315ac1422] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:24.663 INFO 2458 --- [ocp-async-5,fe49aa66328c8815,7033b338ccee8ced] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:25.773 INFO 2458 --- [ocp-schedules-15,7fac4b6a3791bde4,238d3d7e85327abd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:25.777 INFO 2458 --- [ocp-schedules-15,7fac4b6a3791bde4,238d3d7e85327abd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:26.215 INFO 2458 --- [ocp-schedules-14,6971f577f2d7a772,52a58b21df975c7a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:26.267 WARN 2458 --- [scanner-112211836-1,6971f577f2d7a772,d71cc10821e64181] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:26.506 INFO 2458 --- [alarm-detect-8,6971f577f2d7a772,0e448cdedba13d86] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536040 ~ 1761536056 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536055, value=60.594)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536055, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536055=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:34:26.663 WARN 2458 --- [ocp-async-8,52c65227b1d279fe,588cb083ed3f9e68] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:26.663 INFO 2458 --- [ocp-async-4,94b48a8a19c7ed48,aeed2e7922e293ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:26.707 INFO 2458 --- [prometheus-minute-schedule-1,6516dd19fc63c068,1fd27fe60f6b4587] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:26.715 INFO 2458 --- [prometheus-minute-schedule-1,6516dd19fc63c068,1fd27fe60f6b4587] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:27.098 INFO 2458 --- [ocp-schedules-18,253ecde96f14998c,7488bced434ef72f] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:27.639 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:27.640 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:27.739 INFO 2458 --- [ocp-schedules-17,6b8b76ed5df0efb8,cad50d06078a22fc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:34:27.747 INFO 2458 --- [ocp-schedules-17,6b8b76ed5df0efb8,cad50d06078a22fc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:27.747 INFO 2458 --- [ocp-schedules-17,6b8b76ed5df0efb8,cad50d06078a22fc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:28.541 INFO 2458 --- [ocp-async-4,d004fdb74c62a665,8737e9e2638e6b86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:28.663 WARN 2458 --- [ocp-async-10,5d3f6122fcb320ac,51ab1c27a6d1a819] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:28.663 INFO 2458 --- [ocp-async-10,61b124f5c9a8a56f,ba5a30ecf1740987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:29.069 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:34:29.069 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:34:29.071 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:34:29.071 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:34:29.072 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:34:29.072 INFO 2458 --- [ocp-schedules-23,069cbfe75058e4b0,2d213a15c4840eba] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:34:29.509 INFO 2458 --- [ocp-schedules-22,d97ddc940fa9e5c3,5612918b27b7134c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:34:29.522 INFO 2458 --- [ocp-schedules-22,d97ddc940fa9e5c3,5612918b27b7134c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:34:29.522 INFO 2458 --- [ocp-schedules-22,d97ddc940fa9e5c3,5612918b27b7134c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:34:29.707 INFO 2458 --- [prometheus-minute-schedule-1,fce4b8f20f6f2308,4b2ac475fbb26234] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:29.717 INFO 2458 --- [prometheus-minute-schedule-1,fce4b8f20f6f2308,4b2ac475fbb26234] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:30.640 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:30.641 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:30.663 WARN 2458 --- [ocp-async-3,e30c90a892481b56,84ebce3ddbe02548] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:30.663 INFO 2458 --- [ocp-async-9,fce04e5afd8e7c6f,512c8866a4e59437] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:31.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:31.716783+08:00, lastProcessTime=2025-10-27T11:34:21.715066+08:00 2025-10-27 11:34:32.289 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:31.716783+08:00 2025-10-27 11:34:32.663 WARN 2458 --- [ocp-async-7,277a441d337ad546,072030a53eefec45] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:32.663 INFO 2458 --- [ocp-async-4,bb2d7510375adb03,d78b3e42100ce43f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:32.707 INFO 2458 --- [prometheus-minute-schedule-1,35c601f7d8b8ce8c,a19ea70781376048] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:32.716 INFO 2458 --- [prometheus-minute-schedule-1,35c601f7d8b8ce8c,a19ea70781376048] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:33.641 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:33.643 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:33.981 INFO 2458 --- [ocp-schedules-12,91da2365d9ba2843,7d473b8c2c965146] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:34:33.981 INFO 2458 --- [ocp-schedules-12,91da2365d9ba2843,7d473b8c2c965146] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:34:34.664 WARN 2458 --- [ocp-async-2,69d47853267debcb,8bcdf59a4c3f2a20] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:34.664 INFO 2458 --- [ocp-async-2,c3fabf6ef0cee183,2b174f8328b52e85] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:35.707 INFO 2458 --- [prometheus-minute-schedule-1,597ccbfe7032d519,32b7a4f1cc500961] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:35.717 INFO 2458 --- [prometheus-minute-schedule-1,597ccbfe7032d519,32b7a4f1cc500961] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:35.785 INFO 2458 --- [ocp-schedules-27,8a3e00d428045a80,7f7dfce6d52278a5] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:35.788 INFO 2458 --- [ocp-schedules-27,8a3e00d428045a80,7f7dfce6d52278a5] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:36.511 INFO 2458 --- [ocp-schedules-25,0a2158a49691e86d,1415f86e05c8b0e5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:36.543 WARN 2458 --- [scanner-112212060-1,0a2158a49691e86d,02847e20ce172813] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:36.643 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:36.644 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:36.664 WARN 2458 --- [ocp-async-10,06589fdcfd74496b,77686c7909177a93] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:36.664 INFO 2458 --- [ocp-async-3,e8644789cedbcefb,b7782e7aed0f9ea1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:37.106 INFO 2458 --- [ocp-schedules-26,ebc3f53416b1e2e7,f4761c0ad8c036e7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:37.708 INFO 2458 --- [prometheus-cache-monitor1,65c78d0cd4136304,df5b75237b084ad8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20315,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5078,"seriesIdKeySize":8774,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:34:37.708 INFO 2458 --- [prometheus-cache-monitor1,65c78d0cd4136304,df5b75237b084ad8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:34:38.541 INFO 2458 --- [ocp-async-10,9e3657dded476b23,b672b36f89c8e987] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:38.664 WARN 2458 --- [ocp-async-8,a930ad92a0b8bd46,3383712845a86da7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:38.665 INFO 2458 --- [ocp-async-4,2906dd1f998b2686,e47c0e2f8c0405f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:38.707 INFO 2458 --- [prometheus-minute-schedule-1,89f38e27efc99ad6,fbe9bd6a4ddf0180] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:38.715 INFO 2458 --- [prometheus-minute-schedule-1,89f38e27efc99ad6,fbe9bd6a4ddf0180] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:38.842 INFO 2458 --- [ocp-schedules-21,3f83f17d70b153c3,9ed8d8160d5d597e] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:34:38.848 INFO 2458 --- [ocp-schedules-21,3f83f17d70b153c3,9ed8d8160d5d597e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:38.849 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:34:38.849 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:34:38.849 INFO 2458 --- [ocp-schedules-21,3f83f17d70b153c3,9ed8d8160d5d597e] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:34:39.644 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:39.645 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:40.009 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:34:40.009 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:34:40.042 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:40.042 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:34:40.043 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:34:40.043 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:34:40.043 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:34:40.043 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:34:40.053 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:40.057 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:40.057 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:40.058 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:40.058 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:40.059 INFO 2458 --- [Check obproxy1,6b55e9a486766fb6,1bf4384f04d16709] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:34:40.059 INFO 2458 --- [Check obproxy1,6b55e9a486766fb6,1bf4384f04d16709] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:34:40.060 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:40.061 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:40.061 INFO 2458 --- [Collect obproxy1,d0430871df1995ec,6a1eb33ed0343de5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:34:40.061 INFO 2458 --- [Collect obproxy1,d0430871df1995ec,6a1eb33ed0343de5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:34:40.061 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:40.062 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:40.062 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:40.093 INFO 2458 --- [Check obproxy1,6b55e9a486766fb6,1bf4384f04d16709] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:34:40.097 INFO 2458 --- [Check obproxy1,6b55e9a486766fb6,1bf4384f04d16709] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:34:40.097 INFO 2458 --- [Check obproxy1,6b55e9a486766fb6,1bf4384f04d16709] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:34:40.099 INFO 2458 --- [Collect obproxy1,d0430871df1995ec,6a1eb33ed0343de5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:34:40.102 INFO 2458 --- [Collect obproxy1,d0430871df1995ec,6a1eb33ed0343de5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:34:40.105 INFO 2458 --- [Collect obproxy1,d0430871df1995ec,6a1eb33ed0343de5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:34:40.126 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:34:40.129 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:34:40.134 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:34:40.136 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:34:40.137 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:34:40.143 INFO 2458 --- [ocp-schedules-30,f99a6b70d848b874,3e1faf839f7dea9e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:34:40.363 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:34:40.363 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:34:40.468 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:40.469 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:34:40.483 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:40.485 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:34:40.546 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:40.547 INFO 2458 --- [ocp-schedules-10,80ce7b926825fd35,88ca24a581462424] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:34:40.664 WARN 2458 --- [ocp-async-6,14f00c68703d1d3b,5afc65ebd514167f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:40.665 INFO 2458 --- [ocp-async-2,ac1cac07e4088473,4c70a6108d98faa6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:41.289 INFO 2458 --- [ocp-schedules-16,93452cd349cda8ce,126b590c875f52b2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:34:41.707 INFO 2458 --- [prometheus-minute-schedule-1,164cfdbaf702c0e9,36a58d2439920c5d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:41.717 INFO 2458 --- [prometheus-minute-schedule-1,164cfdbaf702c0e9,36a58d2439920c5d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:41.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:41.716472+08:00, lastProcessTime=2025-10-27T11:34:31.716783+08:00 2025-10-27 11:34:42.246 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:41.716472+08:00 2025-10-27 11:34:42.645 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:42.646 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:42.664 WARN 2458 --- [ocp-async-5,2e9932113b07eec2,309a87e67bb76c14] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:42.665 INFO 2458 --- [ocp-async-10,f31dfcbd6af942d8,80a392b0d61e350f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:44.292 INFO 2458 --- [ocp-schedules-30,64c85e609344e4c9,857a00651dc7e301] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:34:44.665 WARN 2458 --- [ocp-async-8,964dac9066b7e807,83d9e3ed18cff532] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:44.665 INFO 2458 --- [ocp-async-7,d7568998e55d8e6b,76d0436c15cb8927] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:44.707 INFO 2458 --- [prometheus-minute-schedule-1,5b04dc7d17a32cef,1c25f83b3c4bb6c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:44.714 INFO 2458 --- [prometheus-minute-schedule-1,5b04dc7d17a32cef,1c25f83b3c4bb6c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:45.102 INFO 2458 --- [prometheus-second-schedule-1,348e3a5bae2bf436,f853d14023c0432d] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:34:45.646 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:45.647 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:45.796 INFO 2458 --- [ocp-schedules-28,d2e5173d57d44732,e52dbdc9e3dbb61f] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:45.799 INFO 2458 --- [ocp-schedules-28,d2e5173d57d44732,e52dbdc9e3dbb61f] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:46.665 WARN 2458 --- [ocp-async-1,4ab4fb60389dba7d,0b2e905ee12c7df1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:46.665 INFO 2458 --- [ocp-async-6,86f05594e1a7106c,b9a0d463ca0dc248] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:46.701 INFO 2458 --- [ocp-schedules-1,fbda7e878d8018fd,5d081199d1767808] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:46.787 WARN 2458 --- [scanner-112212256-1,fbda7e878d8018fd,03419eea08b54dcc] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:46.824 INFO 2458 --- [alarm-detect-18,fbda7e878d8018fd,1aca186595764ed8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536016 ~ 1761536076 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536040, value=204374.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204374.56, values={1761536040=204374.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:34:46.887 INFO 2458 --- [alarm-detect-3,fbda7e878d8018fd,566ac3cd7ca1382f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536016 ~ 1761536076 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536040, value=93.223)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.223, values={1761536040=93.223}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:34:46.906 INFO 2458 --- [alarm-detect-9,fbda7e878d8018fd,6867ca65b0e9da8c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536016 ~ 1761536076 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536040, value=204375.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204375.912, values={1761536040=204375.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:34:46.910 INFO 2458 --- [alarm-detect-1,fbda7e878d8018fd,ceef216f964ba2ad] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536055 ~ 1761536076 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536070, value=48.167)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536070, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536070=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:34:47.002 INFO 2458 --- [ocp-schedules-25,5b0f004908fed390,f9b5402a1220f8c1] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:34:47.002 INFO 2458 --- [ocp-schedules-25,5b0f004908fed390,f9b5402a1220f8c1] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:34:47.062 INFO 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:34:47.062 INFO 2458 --- [pool-882-thread-2,5b0f004908fed390,00f75fcd7acce7f1] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:34:47.109 INFO 2458 --- [pool-882-thread-2,5b0f004908fed390,00f75fcd7acce7f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:47.112 INFO 2458 --- [pool-882-thread-2,5b0f004908fed390,00f75fcd7acce7f1] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535786000000, 1761536087000000] 2025-10-27 11:34:47.114 INFO 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:47.115 INFO 2458 --- [ocp-schedules-12,8c9c88fd8ff03489,1df40ae55c459495] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:47.117 INFO 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761535786000000, 1761536087000000] 2025-10-27 11:34:47.157 WARN 2458 --- [pool-882-thread-2,5b0f004908fed390,00f75fcd7acce7f1] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:34:47.171 INFO 2458 --- [pool-882-thread-2,5b0f004908fed390,00f75fcd7acce7f1] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:34:47.225 WARN 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:34:47.225 WARN 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:34:47.225 WARN 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:34:47.232 INFO 2458 --- [pool-882-thread-1,5b0f004908fed390,ff9bff46c0c23a2b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:34:47.234 INFO 2458 --- [ocp-schedules-25,5b0f004908fed390,f9b5402a1220f8c1] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:34:47.707 INFO 2458 --- [prometheus-minute-schedule-1,eed73ef587b4e7c8,bd792db7cde50801] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:47.715 INFO 2458 --- [prometheus-minute-schedule-1,eed73ef587b4e7c8,bd792db7cde50801] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:47.750 INFO 2458 --- [ocp-schedules-20,b4e7c62f1a32e254,6c44f4d136aeedeb] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:34:47.761 INFO 2458 --- [ocp-schedules-20,b4e7c62f1a32e254,6c44f4d136aeedeb] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:47.761 INFO 2458 --- [ocp-schedules-20,b4e7c62f1a32e254,6c44f4d136aeedeb] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:34:48.541 INFO 2458 --- [ocp-async-4,e8e250c5af6b9c20,b6e0ba30762c5c86] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:48.647 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:48.648 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:48.665 WARN 2458 --- [ocp-async-3,b7473bf7773eb90c,14c74bfed2ffa26e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:48.665 INFO 2458 --- [ocp-async-10,db97abf218234a98,23c5cc7ecef07e0f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:49.566 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,796fc0782e35086a,2e7c0df7ceaa14e6] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=796fc0782e35086a, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:34:49.566 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-10,796fc0782e35086a,2e7c0df7ceaa14e6] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=796fc0782e35086a, duration=1 ms] 2025-10-27 11:34:49.944 INFO 2458 --- [ocp-schedules-22,fe1226f221f5e949,fd4cf01fcefdf2dc] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:34:49.944+08:00, clusters=[1, 2] 2025-10-27 11:34:49.963 INFO 2458 --- [ocp-schedules-22,fe1226f221f5e949,fd4cf01fcefdf2dc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:34:49.966 INFO 2458 --- [ocp-schedules-22,fe1226f221f5e949,fd4cf01fcefdf2dc] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:34:50.665 WARN 2458 --- [ocp-async-8,461b742f948c16cd,a428499bb0d4c259] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:50.665 INFO 2458 --- [ocp-async-7,46560b3c66aac4d6,534c63ddc3c0a5f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:50.707 INFO 2458 --- [prometheus-minute-schedule-1,0c18fc0f795fa22f,1a12cdf197953cb7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:50.714 INFO 2458 --- [prometheus-minute-schedule-1,0c18fc0f795fa22f,1a12cdf197953cb7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:51.410 INFO 2458 --- [ocp-schedules-24,1bd294103781803f,bb8eb5273a14f8c8] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:51.410 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:34:51.410 INFO 2458 --- [ocp-schedules-9,f4913a05b4c277b3,633e99e40f529104] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:51.410 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:34:51.410 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:34:51.410 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:34:51.410 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:34:51.414 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:34:51.415 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:34:51.415 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:34:51.415 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:34:51.415 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:34:51.425 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:51.425 INFO 2458 --- [ocp-schedules-22,29fbae9e8e840b42,3cb2b73a40316b96] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:34:51.426 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:51.427 INFO 2458 --- [host-check-clockdiff1,29fbae9e8e840b42,7eabaea0707a37c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:34:51.427 INFO 2458 --- [host-check-clockdiff2,29fbae9e8e840b42,b2e51a606f2bb5aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:34:51.427 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:51.427 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:51.427 INFO 2458 --- [host-check-clockdiff3,29fbae9e8e840b42,796ce6b7cbee46aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:34:51.428 INFO 2458 --- [host-check-clockdiff4,29fbae9e8e840b42,eed854fb88da830e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:34:51.428 INFO 2458 --- [host-check-clockdiff5,29fbae9e8e840b42,aa65532354bd06c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:34:51.428 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:51.429 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:51.430 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:51.431 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:51.431 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:51.431 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:51.431 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:51.432 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:51.432 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:51.437 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:51.437 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:51.442 INFO 2458 --- [ocp-schedules-25,f4494e6c03fb33fe,0bf8f32726341da0] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:34:51.446 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:51.437 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:51.446 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:51.447 INFO 2458 --- [host-check-clockdiff5,29fbae9e8e840b42,aa65532354bd06c9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536091 0 0 , error= 2025-10-27 11:34:51.449 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:51.449 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:51.453 INFO 2458 --- [host-check-clockdiff1,29fbae9e8e840b42,7eabaea0707a37c4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536091 0 0 , error= 2025-10-27 11:34:51.527 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:34:51.529 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:34:51.530 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:34:51.531 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:34:51.531 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:34:51.531 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:34:51.530442987+08:00 2025-10-27 11:34:51.531 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:34:51.531+08:00 2025-10-27 11:34:51.531 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-agent-status1,f4913a05b4c277b3,20ef227e81970f93] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:34:51.532568132+08:00 2025-10-27 11:34:51.532 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:34:51.531734318+08:00 2025-10-27 11:34:51.533 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:34:51.532+08:00 2025-10-27 11:34:51.533 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:34:51.533+08:00 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status4,f4913a05b4c277b3,632872d6e44e2d5f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-agent-status3,f4913a05b4c277b3,363d85b252ebd3f6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:34:51.533869693+08:00 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:34:51.534 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:34:51.534+08:00 2025-10-27 11:34:51.535 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:34:51.534888719+08:00 2025-10-27 11:34:51.535 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:34:51.535+08:00 2025-10-27 11:34:51.536 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:34:51.536 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:34:51.536 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:34:51.536 INFO 2458 --- [host-check-agent-status5,f4913a05b4c277b3,a3a22c0bc2cc00e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:34:51.539 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:34:51.539 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:34:51.539 INFO 2458 --- [host-check-agent-status2,f4913a05b4c277b3,55556d5d272c3c38] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:34:51.540 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:34:51.531+08:00 and status AVAILABLE 2025-10-27 11:34:51.541 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:34:51.532+08:00 and status AVAILABLE 2025-10-27 11:34:51.541 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:34:51.533+08:00 and status AVAILABLE 2025-10-27 11:34:51.542 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:34:51.534+08:00 and status AVAILABLE 2025-10-27 11:34:51.542 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:34:51.535+08:00 and status AVAILABLE 2025-10-27 11:34:51.545 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:34:51.545 INFO 2458 --- [ocp-schedules-17,41565c8bf873f5a1,e93799fc60076883] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:34:51.547 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:34:51.547 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:34:51.547 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:34:51.547 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:34:51.549 INFO 2458 --- [ocp-schedules-12,cfd38c54958a4712,0c3614733e31f881] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:34:51.550 INFO 2458 --- [ocp-schedules-17,41565c8bf873f5a1,e93799fc60076883] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:51.553 INFO 2458 --- [ocp-schedules-12,cfd38c54958a4712,0c3614733e31f881] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:51.554 INFO 2458 --- [ocp-schedules-8,fd2429a38010ceab,c1a538738e80f830] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:34:51.554 INFO 2458 --- [ocp-schedules-12,cfd38c54958a4712,0c3614733e31f881] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:34:51.558 INFO 2458 --- [ocp-schedules-8,fd2429a38010ceab,c1a538738e80f830] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:34:51.566 INFO 2458 --- [ocp-schedules-11,9b49206a0ed5e380,9f65afdc508fc301] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:34:51.568 INFO 2458 --- [ocp-schedules-20,6d5e7e31d169be2b,a59bbd17f38a72c3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:34:51.569 INFO 2458 --- [ocp-schedules-5,2adbe31d0475b62a,afffdb302ff4bf4e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:34:51.572 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:51.572 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:34:51.575 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:51.575 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:51.575 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:34:51.577 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:34:51.578 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:51.578 INFO 2458 --- [ocp-schedules-20,6d5e7e31d169be2b,a59bbd17f38a72c3] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:34:51.580 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:34:51.581 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:34:51.583 INFO 2458 --- [ocp-schedules-5,2adbe31d0475b62a,afffdb302ff4bf4e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:34:51.584 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:34:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:51.602 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:34:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.638 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:51.648 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:51.649 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:51.651 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.652 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:34:51.652 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:51.652 INFO 2458 --- [ocp-schedules-17,41565c8bf873f5a1,e93799fc60076883] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:34:51.659 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:51.659 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:51.663 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:51.666 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:51.676 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:34:51.682 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:34:51.683 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:34:51.690 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:34:51.693 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:34:51.714 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:34:51.717 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.717 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:51.723 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:34:51.722196+08:00, lastProcessTime=2025-10-27T11:34:41.716472+08:00 2025-10-27 11:34:51.724 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:51.724 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.725 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:34:51.726 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:34:51.730 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:34:51.731 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:34:51.735 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:34:51.751 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:34:51.759 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:51.762 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:51.766 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:51.771 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:51.774 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:34:51.779 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:34:51.782 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:34:51.787 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:34:51.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:51.789 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.790 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:51.791 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.795 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:34:51.797 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:51.801 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:34:51.802 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:34:51.805 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:51.806 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.806 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:34:51.814 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:34:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:51.830 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:34:51.831 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:34:51.838 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:34:51.841 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:34:51.850 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:51.853 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:51.860 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:51.864 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:51.867 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:34:51.896 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:34:51.896 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:34:51.897 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:51.897 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:34:51.901 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:51.903 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:34:51.911 INFO 2458 --- [ocp-schedules-30,d3d96332fd006030,d3e4a0ddec7ac608] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:34:51.917 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:34:51.925 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:34:51.926 INFO 2458 --- [host-check-host-status3,1bd294103781803f,0bce2e82fdb6c69d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:34:51.927 INFO 2458 --- [host-check-host-status4,1bd294103781803f,2d967748efa8e59f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:34:51.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:51.928 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.929 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:34:51.931 INFO 2458 --- [host-check-host-status5,1bd294103781803f,067b4979c7503326] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:34:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:51.934 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:34:51.935 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:34:51.936 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:34:51.936 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:51.939 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:34:51.947 INFO 2458 --- [host-check-host-status2,1bd294103781803f,e32531ba03b99e88] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:34:51.966 INFO 2458 --- [host-check-host-status1,1bd294103781803f,610ae0d6be0fab9f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:34:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.005 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:52.008 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:34:52.029 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:34:51.722196+08:00 2025-10-27 11:34:52.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:52.053 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:34:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.054 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:34:52.056 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.057 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:34:52.074 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:34:52.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:52.099 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:52.099 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.100 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:34:52.125 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:34:52.125 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:34:52.128 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:34:52.163 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.163 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:34:52.200 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:34:52.240 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:52.241 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.242 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:34:52.246 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:34:52.257 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:34:52.266 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:34:52.269 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:34:52.317 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:34:52.383 INFO 2458 --- [host-check-clockdiff2,29fbae9e8e840b42,b2e51a606f2bb5aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536092 -1 -1 , error= 2025-10-27 11:34:52.383 INFO 2458 --- [host-check-clockdiff4,29fbae9e8e840b42,eed854fb88da830e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536092 0 0 , error= 2025-10-27 11:34:52.384 INFO 2458 --- [host-check-clockdiff3,29fbae9e8e840b42,796ce6b7cbee46aa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536092 0 0 , error= 2025-10-27 11:34:52.384 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:34:52.384 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.385 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:34:52.387 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.388 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:34:52.425 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:34:52.465 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:34:52.466 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:34:52.466 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:34:52.511 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:34:52.512 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:34:52.512 INFO 2458 --- [ocp-schedules-8,fd2429a38010ceab,c1a538738e80f830] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:34:52.665 WARN 2458 --- [ocp-async-4,c1cda0aac86f496f,f2c8925beff2cdd0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:52.666 INFO 2458 --- [ocp-async-2,684d0165d32ca1da,8e9bbe4d4fa35373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:34:53.707 INFO 2458 --- [prometheus-minute-schedule-1,6f7485273f3d9c29,bc088d044b0bd338] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:53.715 INFO 2458 --- [prometheus-minute-schedule-1,6f7485273f3d9c29,bc088d044b0bd338] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:54.649 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:54.650 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:54.665 WARN 2458 --- [ocp-async-5,edebf7dc9a0398e3,d9602bc84b31ce02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:54.666 INFO 2458 --- [ocp-async-8,ed600e4bdb290dd0,c61a327f256d0bff] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:55.709 INFO 2458 --- [ocp-schedules-24,a23670f305df990b,7a180816a10fe9ac] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:34:55.834 INFO 2458 --- [ocp-schedules-21,c62508dd4b5eea13,7e65890b7f42fd19] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:34:55.838 INFO 2458 --- [ocp-schedules-21,c62508dd4b5eea13,7e65890b7f42fd19] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:34:56.665 WARN 2458 --- [ocp-async-8,8673f68adf796d72,1f6df7f685b4764f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:56.666 INFO 2458 --- [ocp-async-1,82293ceb51852e2d,faaa8a3883638105] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:56.707 INFO 2458 --- [prometheus-minute-schedule-1,3c4538195d43af1a,4c344a0a6ff90be3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:56.714 INFO 2458 --- [prometheus-minute-schedule-1,3c4538195d43af1a,4c344a0a6ff90be3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:34:56.917 INFO 2458 --- [ocp-schedules-29,7d66239644d9414d,e7587c3f32ee55bd] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:34:56.927 WARN 2458 --- [scanner-112212483-1,7d66239644d9414d,308400e62bdcf8e0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:34:57.049 INFO 2458 --- [alarm-detect-8,7d66239644d9414d,f286c351583c5c95] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536070 ~ 1761536087 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536085, value=51.449)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536085, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536085=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:34:57.121 INFO 2458 --- [ocp-schedules-14,7f4bc6d412d202c8,c1b1541505749ea4] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:34:57.650 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:34:57.651 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:34:58.542 INFO 2458 --- [ocp-async-7,6ef7206296ce7468,e5f3cefc3c3c86a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:34:58.666 WARN 2458 --- [ocp-async-4,bfb64c99bf6a8242,1d2783a74648475a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:34:58.666 INFO 2458 --- [ocp-async-6,c8d16a08909675ea,5bcbd62cbca4949d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:34:59.562 INFO 2458 --- [ocp-schedules-10,d4e37b6a91bbcf6c,877aac0a9caaa7d8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:34:59.707 INFO 2458 --- [prometheus-minute-schedule-1,92add5801433e084,6df87b335f962d14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536100, interval=60 2025-10-27 11:34:59.714 INFO 2458 --- [prometheus-minute-schedule-1,92add5801433e084,6df87b335f962d14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:00.545 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:30:01.0, nextRun=2025-10-27 11:35:00.0, currentDate=Mon Oct 27 11:35:00 CST 2025 2025-10-27 11:35:00.545 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:30:01.0, nextRunTime=2025-10-27 11:35:00.0} 2025-10-27 11:35:00.558 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:35:00.558 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:35:00.558+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:35:00.580 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:30:01.0, nextRun=2025-10-27 11:35:00.0, currentDate=Mon Oct 27 11:35:00 CST 2025 2025-10-27 11:35:00.580 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:30:01.0, nextRunTime=2025-10-27 11:35:00.0} 2025-10-27 11:35:00.640 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:35:00.648 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:35:00.651 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:00.652 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:00.653 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:35:00.653 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,3a523cf5cc623f87] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:35:00.653+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:35:00.666 WARN 2458 --- [ocp-async-9,d459f6f2d2119ca2,dd07499483e76574] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:00.666 INFO 2458 --- [ocp-async-2,aa48c58a094de07e,2187ee8b7340ee8f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:01.163 INFO 2458 --- [subtask-coordinator1,8964d58b78e8b300,1cd91925a0f34dae] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9179, operation:EXECUTE, state: RUNNING 2025-10-27 11:35:01.178 INFO 2458 --- [subtask-coordinator1,8964d58b78e8b300,1cd91925a0f34dae] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9180, operation:EXECUTE, state: RUNNING 2025-10-27 11:35:01.199 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:35:01.200 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9179, context=Context{parallelIdx=-1, stringMap={task_instance_id=9111, task_operation=execute, latest_execution_start_time=2025-10-27T11:35:01.150+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9179}, listMap={}}, executor=10.40.65.146 2025-10-27 11:35:01.215 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:35:01.219 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9180, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9112, task_operation=execute, latest_execution_start_time=2025-10-27T11:35:01.174+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9180}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:35:01.219 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:35:01.273 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:01.356 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:35:01.357 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:01.359 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:35:01.422 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:01.442 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:35:01.445 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:35:01.514 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:01.549 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:35:01.651 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:01.713908+08:00, lastProcessTime=2025-10-27T11:34:51.722196+08:00 2025-10-27 11:35:01.778 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:02.002 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:01.713908+08:00 2025-10-27 11:35:02.019 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:35:02.019 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:35:02.034 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:02.095 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:35:02.127 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:35:02.131 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:35:02.139 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:35:02.139 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:35:02.148 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.150 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,8bfa74eb495ed830] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9180, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:35:02.169 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.190 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.218 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.274 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.325 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.455 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.491 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:35:02.500 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:35:02.500 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:35:02.525 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,0eb329367e550d47] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9179, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:35:02.666 WARN 2458 --- [ocp-async-2,6f732a7bfbd19f0b,1ebb8d995fc8312f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:02.666 INFO 2458 --- [ocp-async-7,87d35a80c84b9c70,201b79403a1e41ad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:02.707 INFO 2458 --- [prometheus-minute-schedule-1,2f06383012a8bf68,ee0e9a96e37c4816] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:02.715 INFO 2458 --- [prometheus-minute-schedule-1,2f06383012a8bf68,ee0e9a96e37c4816] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:03.157 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9111, status:SUCCESS 2025-10-27 11:35:03.167 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9111, status:SUCCESS 2025-10-27 11:35:03.178 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9112, status:SUCCESS 2025-10-27 11:35:03.180 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9112, status:SUCCESS 2025-10-27 11:35:03.652 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:03.653 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:04.666 WARN 2458 --- [ocp-async-3,75eec92285989e8c,f263a02fb948c9f4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:04.666 INFO 2458 --- [ocp-async-5,d9452021842d1f88,d8be50fdcda313dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:04.838 INFO 2458 --- [ocp-schedules-6,e8349c4cc919632c,283ce3f54fe3aa94] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:35:04.844 INFO 2458 --- [ocp-schedules-6,e8349c4cc919632c,283ce3f54fe3aa94] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:04.885 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:04.885 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:04.895 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:35:04.909 INFO 2458 --- [ocp-schedules-6,e8349c4cc919632c,283ce3f54fe3aa94] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:35:05.707 INFO 2458 --- [prometheus-minute-schedule-1,dca3c936138a2dc5,904d8ac7c48c4e62] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:05.717 INFO 2458 --- [prometheus-minute-schedule-1,dca3c936138a2dc5,904d8ac7c48c4e62] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:05.846 INFO 2458 --- [ocp-schedules-24,46cdebee33d346ea,938a212968462944] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:05.849 INFO 2458 --- [ocp-schedules-24,46cdebee33d346ea,938a212968462944] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:06.423 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a9f26dae3a5bac03,94f54d479043e9ca] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=a9f26dae3a5bac03, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:35:06.423 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,a9f26dae3a5bac03,94f54d479043e9ca] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=a9f26dae3a5bac03, duration=1 ms] 2025-10-27 11:35:06.653 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:06.655 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:06.666 WARN 2458 --- [ocp-async-6,77f4a94b33427474,cd8e419e84e3408b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:06.667 INFO 2458 --- [ocp-async-8,899f2c30fb059d40,c91b230b411f6c92] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:07.055 INFO 2458 --- [ocp-schedules-20,14d6b63ebaea1f1e,a7c3ece108b7fddb] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:07.120 WARN 2458 --- [scanner-112212903-1,14d6b63ebaea1f1e,16ee4953c35db37d] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:07.126 INFO 2458 --- [ocp-schedules-29,bfc160731a6372fa,3d54470a5d500132] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:07.764 INFO 2458 --- [ocp-schedules-25,781d2a51a96745b9,ad96e0410e099b25] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:35:07.772 INFO 2458 --- [ocp-schedules-25,781d2a51a96745b9,ad96e0410e099b25] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:07.772 INFO 2458 --- [ocp-schedules-25,781d2a51a96745b9,ad96e0410e099b25] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:08.542 INFO 2458 --- [ocp-async-8,d0d3d8bc04aacea8,0e201bbe24935b95] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:08.666 WARN 2458 --- [ocp-async-1,b5f93099b4c4c7cd,4f9bc2b1aab4b9d6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:08.667 INFO 2458 --- [ocp-async-7,21aaeefa76d39340,26e80fd7aeff08a3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:08.707 INFO 2458 --- [prometheus-minute-schedule-1,a73f7ff57c480757,b84d18580ed68c54] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:08.713 INFO 2458 --- [prometheus-minute-schedule-1,a73f7ff57c480757,b84d18580ed68c54] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:09.655 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:09.656 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:10.007 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:35:10.008 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:35:10.042 INFO 2458 --- [Check obproxy1,8134603188f942b0,46f56c05678ef741] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:35:10.042 INFO 2458 --- [Check obproxy1,8134603188f942b0,46f56c05678ef741] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:35:10.051 INFO 2458 --- [Collect obproxy1,ed394c40cb6edf04,9dad8a44d016899d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:35:10.051 INFO 2458 --- [Collect obproxy1,ed394c40cb6edf04,9dad8a44d016899d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:35:10.070 INFO 2458 --- [Check obproxy1,8134603188f942b0,46f56c05678ef741] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:35:10.073 INFO 2458 --- [Check obproxy1,8134603188f942b0,46f56c05678ef741] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:35:10.073 INFO 2458 --- [Check obproxy1,8134603188f942b0,46f56c05678ef741] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:35:10.081 INFO 2458 --- [Collect obproxy1,ed394c40cb6edf04,9dad8a44d016899d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:35:10.084 INFO 2458 --- [Collect obproxy1,ed394c40cb6edf04,9dad8a44d016899d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:35:10.087 INFO 2458 --- [Collect obproxy1,ed394c40cb6edf04,9dad8a44d016899d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:35:10.176 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:35:10.666 WARN 2458 --- [ocp-async-3,927ef35ddb3ac9a8,48de4e224945b634] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:10.667 INFO 2458 --- [ocp-async-9,d06f74a9fd943d66,56583d925b9c2fde] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:10.683 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,8088c1f54e1eef72,4573eb68a7ae24a0] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/software-packages?null, client=10.40.198.13, traceId=8088c1f54e1eef72, method=SuccessResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(MultipartFile), args=] 2025-10-27 11:35:10.728 ERROR 2458 --- [http-nio-0.0.0.0-8081-exec-2,8088c1f54e1eef72,4573eb68a7ae24a0] c.o.ocp.core.util.ExceptionUtils : Checked Exception: com.oceanbase.ocp.core.exception.IllegalArgumentException occurred with code error.file.software.exists, and args [obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm] 2025-10-27 11:35:10.728 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-2,8088c1f54e1eef72,4573eb68a7ae24a0] c.o.o.s.c.trace.RequestTracingAspect : API Error: [POST /api/v2/software-packages client=10.40.198.13, traceId=8088c1f54e1eef72, args=, duration=87540 ms, errorMsg=[OCP IllegalArgumentException]: status=400 BAD_REQUEST, errorCode=FILE_SOFTWARE_EXISTS, args=obbinlog-ce-4.0.1-100000162024122511.el7.x86_64.rpm] 2025-10-27 11:35:10.749 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=1000, affectRows=1000 2025-10-27 11:35:10.755 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=994, validCount=992 2025-10-27 11:35:10.816 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=1000, affectRows=1000 2025-10-27 11:35:10.820 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=999, validCount=995 2025-10-27 11:35:10.845 ERROR 2458 --- [metric-archive-second-1,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134481423) Duplicate entry '835014129065861120-1761535980' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134481423) Duplicate entry '835014129065861120-1761535980' for key 'PRIMARY' 2025-10-27 11:35:11.650 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=1000, affectRows=1000 2025-10-27 11:35:11.655 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=996, validCount=992 2025-10-27 11:35:11.657 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570103) Duplicate entry '835370602954432512-1761535980' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570103) Duplicate entry '835370602954432512-1761535980' for key 'PRIMARY' 2025-10-27 11:35:11.677 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=78, affectRows=78 2025-10-27 11:35:11.678 INFO 2458 --- [metric-archive-second-2,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=77, validCount=77 2025-10-27 11:35:11.707 INFO 2458 --- [prometheus-minute-schedule-1,f7982cbc17d537a6,e2ab56e31fd4afc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:11.714 INFO 2458 --- [prometheus-minute-schedule-1,f7982cbc17d537a6,e2ab56e31fd4afc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:11.714152+08:00, lastProcessTime=2025-10-27T11:35:01.713908+08:00 2025-10-27 11:35:11.918 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:11.714152+08:00 2025-10-27 11:35:12.656 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:12.657 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:12.667 WARN 2458 --- [ocp-async-6,cf5135027fc604e6,d48bcee358fca41a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:12.667 INFO 2458 --- [ocp-async-8,52eeecc6c3bdcbd7,01fd9958b6835bcd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:12.790 INFO 2458 --- [metric-archive-async-6,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536040, nodeCount=821, affectRows=821 2025-10-27 11:35:14.667 WARN 2458 --- [ocp-async-2,22d6bf63d254fbcb,f38e6ff82e3e633d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:14.667 INFO 2458 --- [ocp-async-10,48bfed3f0ea6b2af,d6e0f01b00a7a628] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:14.707 INFO 2458 --- [prometheus-minute-schedule-1,4e5436ed360d61dd,7aa2871f372a7594] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:14.713 INFO 2458 --- [prometheus-minute-schedule-1,4e5436ed360d61dd,7aa2871f372a7594] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:15.396 INFO 2458 --- [ocp-schedules-18,4fef65bf3c40032d,af48805abc5ddc2e] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:35:15.403 INFO 2458 --- [ocp-schedules-18,4fef65bf3c40032d,af48805abc5ddc2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:15.429 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=1000, affectRows=1000 2025-10-27 11:35:15.433 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=996, validCount=994 2025-10-27 11:35:15.495 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:35:15.495 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.499 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:35:15.507 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.508 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.534 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.535 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.548 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:35:15.548 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.553 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:35:15.561 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.562 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.564 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.564 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.570 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.570 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.590 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.590 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.598 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.599 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.618 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.619 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.629 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.629 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:35:15.633 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.635 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.657 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:15.658 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:15.659 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.660 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.661 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.662 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.662 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.663 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.664 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.665 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.666 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.666 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.667 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.667 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.667 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.668 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.669 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:35:15.673 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:35:15.696 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.697 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:35:15.699 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.700 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:35:15.738 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.739 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:35:15.766 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:15.767 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.768 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:35:15.771 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.772 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:35:15.775 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.775 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:35:15.779 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:15.779 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:35:15.779 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:15.779 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:35:15.782 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:35:15.782 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:35:15.801 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.802 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:35:15.832 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.833 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:35:15.857 INFO 2458 --- [ocp-schedules-28,bb7f14b9501c5409,3157c5db778a74cb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:15.862 INFO 2458 --- [ocp-schedules-28,bb7f14b9501c5409,3157c5db778a74cb] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:15.880 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.881 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.881 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.882 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.883 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.884 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.884 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.885 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.886 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.887 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.887 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.888 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.889 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.889 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.890 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.891 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.892 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.893 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.893 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.894 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:35:15.895 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.895 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:35:15.896 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.897 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:35:15.897 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:35:15.900 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:35:15.991 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:15.992 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:15.992 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:35:16.024 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:35:16.029 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:35:16.029 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:35:16.029 INFO 2458 --- [ocp-schedules-18,4fef65bf3c40032d,af48805abc5ddc2e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:35:16.062 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536040, nodeCount=1000, affectRows=1000 2025-10-27 11:35:16.066 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536040, nodeCount=1000, validCount=986 2025-10-27 11:35:16.360 INFO 2458 --- [ocp-schedules-21,5428e56536b78d9e,dd69360f5ec562e0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:35:16.360+08:00, clusters=[1, 2] 2025-10-27 11:35:16.364 INFO 2458 --- [ocp-schedules-21,5428e56536b78d9e,dd69360f5ec562e0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:35:16.368 INFO 2458 --- [ocp-schedules-21,5428e56536b78d9e,dd69360f5ec562e0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:35:16.667 WARN 2458 --- [ocp-async-7,ec2db92901d7ce83,c005759d25319293] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:16.669 INFO 2458 --- [ocp-async-4,67bed7026b11a60e,b8d4d3d7e97051d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:17.135 INFO 2458 --- [ocp-schedules-26,1c60dcc92c6df34c,ce7a32d2746a69ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:17.251 INFO 2458 --- [ocp-schedules-30,b9545d3ab389df8c,b11137981161f0bc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:17.286 WARN 2458 --- [scanner-112214679-1,b9545d3ab389df8c,d6949f8ee3cee8d0] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:17.380 INFO 2458 --- [alarm-detect-11,b9545d3ab389df8c,d977debdd5f3d400] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536085 ~ 1761536107 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=6.2399770016E7)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=17409.0)])]}, metricEvaluateResults={request_queue_time > 1000=[ExpressionEvaluateResult(matchValue=1000.0, metric=request_queue_time, match=true, value=6.2399770016E7, values={1761536100=6.2399770016E7}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})], ob_tenant_request_queue_size > 10=[ExpressionEvaluateResult(matchValue=10.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536100=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:35:17.380 INFO 2458 --- [alarm-detect-11,b9545d3ab389df8c,d977debdd5f3d400] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536085 ~ 1761536107 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=6.2399770016E7)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536100=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:35:17.384 INFO 2458 --- [alarm-detect-11,b9545d3ab389df8c,d977debdd5f3d400] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-2, currentState=Pending 2025-10-27 11:35:17.707 INFO 2458 --- [prometheus-minute-schedule-1,18fb5cf0497e7bb1,2a43c9ff6bf03dc0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:17.714 INFO 2458 --- [prometheus-minute-schedule-1,18fb5cf0497e7bb1,2a43c9ff6bf03dc0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:18.332 INFO 2458 --- [ocp-schedules-4,4e0a88dc472bfec6,8d137cbb7198f44c] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:35:18.332 INFO 2458 --- [ocp-schedules-4,4e0a88dc472bfec6,8d137cbb7198f44c] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:35:18.390 INFO 2458 --- [Collect all tenants major compactions1,4e0a88dc472bfec6,217ab231f08fff9a] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:35:18.390 INFO 2458 --- [Collect all tenants major compactions4,4e0a88dc472bfec6,f4d3f3921a6e50db] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:35:18.390 INFO 2458 --- [Collect all tenants major compactions3,4e0a88dc472bfec6,79ab7cbf8b3abeb0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:35:18.390 INFO 2458 --- [Collect all tenants major compactions2,4e0a88dc472bfec6,73507575722ad02f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:35:18.392 INFO 2458 --- [Collect all tenants major compactions5,4e0a88dc472bfec6,e662b5602bcebb4f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:35:18.392 INFO 2458 --- [Collect all tenants major compactions6,4e0a88dc472bfec6,aab37c1a7185b43e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:35:18.428 INFO 2458 --- [Collect all tenants major compactions6,4e0a88dc472bfec6,aab37c1a7185b43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.429 INFO 2458 --- [Collect all tenants major compactions6,4e0a88dc472bfec6,aab37c1a7185b43e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:35:18.432 INFO 2458 --- [Collect all tenants major compactions5,4e0a88dc472bfec6,e662b5602bcebb4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.433 INFO 2458 --- [Collect all tenants major compactions6,4e0a88dc472bfec6,aab37c1a7185b43e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:35:18.434 INFO 2458 --- [Collect all tenants major compactions5,4e0a88dc472bfec6,e662b5602bcebb4f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:35:18.437 INFO 2458 --- [Collect all tenants major compactions5,4e0a88dc472bfec6,e662b5602bcebb4f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:35:18.443 INFO 2458 --- [Collect all tenants major compactions1,4e0a88dc472bfec6,217ab231f08fff9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.443 INFO 2458 --- [Collect all tenants major compactions3,4e0a88dc472bfec6,79ab7cbf8b3abeb0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.444 INFO 2458 --- [Collect all tenants major compactions1,4e0a88dc472bfec6,217ab231f08fff9a] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:35:18.445 INFO 2458 --- [Collect all tenants major compactions3,4e0a88dc472bfec6,79ab7cbf8b3abeb0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:35:18.448 INFO 2458 --- [Collect all tenants major compactions1,4e0a88dc472bfec6,217ab231f08fff9a] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:35:18.449 INFO 2458 --- [Collect all tenants major compactions4,4e0a88dc472bfec6,f4d3f3921a6e50db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.449 INFO 2458 --- [Collect all tenants major compactions3,4e0a88dc472bfec6,79ab7cbf8b3abeb0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:35:18.450 INFO 2458 --- [Collect all tenants major compactions4,4e0a88dc472bfec6,f4d3f3921a6e50db] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:35:18.453 INFO 2458 --- [Collect all tenants major compactions2,4e0a88dc472bfec6,73507575722ad02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:18.454 INFO 2458 --- [Collect all tenants major compactions2,4e0a88dc472bfec6,73507575722ad02f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:35:18.455 INFO 2458 --- [Collect all tenants major compactions4,4e0a88dc472bfec6,f4d3f3921a6e50db] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:35:18.457 INFO 2458 --- [Collect all tenants major compactions2,4e0a88dc472bfec6,73507575722ad02f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:35:18.459 INFO 2458 --- [ocp-schedules-4,4e0a88dc472bfec6,8d137cbb7198f44c] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:35:18.542 INFO 2458 --- [ocp-async-4,8f2052e5b5858f78,2ec45e2b32801507] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:18.658 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:18.658 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:18.667 WARN 2458 --- [ocp-async-6,bdf38c4344d71f32,9a786f01646a335d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:18.670 INFO 2458 --- [ocp-async-8,7317e2a191fd2e90,ade28b4f375ae59e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:20.669 WARN 2458 --- [ocp-async-1,2bc01d8d5adf00d2,86dea3cf4d0bd3b8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:20.670 INFO 2458 --- [ocp-async-10,c2f2db05b6ef4abb,079be17ecc881f39] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:20.707 INFO 2458 --- [prometheus-minute-schedule-1,387effaca85ef3fe,ded3da19409bfcb1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:20.714 INFO 2458 --- [prometheus-minute-schedule-1,387effaca85ef3fe,ded3da19409bfcb1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:20.757 INFO 2458 --- [ocp-schedules-14,082ad4930c7e4280,44bf835f8cb8efb8] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:35:21.408 INFO 2458 --- [ocp-schedules-10,59cd05ac0d8bf882,2871501f3f7c9a27] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:21.409 INFO 2458 --- [ocp-schedules-30,e91fb82f1ebafe8f,320fb3c6ecbc4a9b] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:35:21.409 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:35:21.410 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:35:21.410 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:35:21.410 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:35:21.411 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:35:21.423 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:21.423 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:21.423 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:21.423 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:21.425 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:21.426 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:21.427 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:21.427 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:21.428 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:21.428 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:21.428 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:21.429 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:21.430 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:21.430 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:21.430 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:21.430 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:21.431 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:21.432 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:21.433 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:21.434 INFO 2458 --- [ocp-schedules-1,ab3447a367963b0e,d425091584c9fea9] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:21.434 INFO 2458 --- [host-check-clockdiff1,ab3447a367963b0e,cdb970556e121a79] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:35:21.434 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:21.434 INFO 2458 --- [host-check-clockdiff2,ab3447a367963b0e,2d891e2cb70901c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:35:21.435 INFO 2458 --- [host-check-clockdiff3,ab3447a367963b0e,b5f98beca8b03a13] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:35:21.435 INFO 2458 --- [host-check-clockdiff4,ab3447a367963b0e,3b1b421f03113aa4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:35:21.435 INFO 2458 --- [host-check-clockdiff5,ab3447a367963b0e,3882d53db5135a4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:35:21.441 INFO 2458 --- [host-check-clockdiff5,ab3447a367963b0e,3882d53db5135a4c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536121 0 0 , error= 2025-10-27 11:35:21.442 INFO 2458 --- [ocp-schedules-17,f96ee4fe3b007b74,4f840b6c5ca6979c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:35:21.455 INFO 2458 --- [host-check-clockdiff3,ab3447a367963b0e,b5f98beca8b03a13] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536121 -1 -1 , error= 2025-10-27 11:35:21.472 INFO 2458 --- [host-check-clockdiff1,ab3447a367963b0e,cdb970556e121a79] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536121 0 0 , error= 2025-10-27 11:35:21.484 INFO 2458 --- [host-check-clockdiff4,ab3447a367963b0e,3b1b421f03113aa4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536121 0 0 , error= 2025-10-27 11:35:21.488 INFO 2458 --- [host-check-clockdiff2,ab3447a367963b0e,2d891e2cb70901c8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536121 -1 -1 , error= 2025-10-27 11:35:21.514 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:35:21.518 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:35:21.518 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:35:21.518 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:35:21.519 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:35:21.519 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:35:21.519 INFO 2458 --- [host-check-agent-status5,e91fb82f1ebafe8f,68f790e8288a7355] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:35:21.520 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:35:21.520297789+08:00 2025-10-27 11:35:21.520 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:35:21.521 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:35:21.520+08:00 2025-10-27 11:35:21.521 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:35:21.520365244+08:00 2025-10-27 11:35:21.521 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:35:21.521+08:00 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status2,e91fb82f1ebafe8f,901eec67d664179d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:35:21.522 INFO 2458 --- [host-check-agent-status3,e91fb82f1ebafe8f,498abce36191cc32] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:35:21.524 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:35:21.524 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:35:21.524235594+08:00 2025-10-27 11:35:21.525 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:35:21.525 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:35:21.525 INFO 2458 --- [host-check-agent-status1,e91fb82f1ebafe8f,f36920e1aea73c05] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:35:21.525 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:35:21.524+08:00 2025-10-27 11:35:21.527 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:35:21.528 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:35:21.528 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:35:21.528875706+08:00 2025-10-27 11:35:21.528 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:35:21.528446662+08:00 2025-10-27 11:35:21.529 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:35:21.528+08:00 2025-10-27 11:35:21.529 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:35:21.529 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:35:21.529+08:00 2025-10-27 11:35:21.529 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:35:21.520+08:00 and status AVAILABLE 2025-10-27 11:35:21.531 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:35:21.521+08:00 and status AVAILABLE 2025-10-27 11:35:21.531 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:35:21.531 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:35:21.531 INFO 2458 --- [host-check-agent-status4,e91fb82f1ebafe8f,3ca2f88612890014] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:35:21.533 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:35:21.524+08:00 and status AVAILABLE 2025-10-27 11:35:21.535 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:35:21.536 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:35:21.528+08:00 and status AVAILABLE 2025-10-27 11:35:21.536 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:35:21.529+08:00 and status AVAILABLE 2025-10-27 11:35:21.537 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:35:21.537 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:35:21.541 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:35:21.543 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:35:21.545 INFO 2458 --- [ocp-schedules-25,0af6248ecb701990,ec2e1cb4780fb582] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:35:21.549 INFO 2458 --- [ocp-schedules-20,1b233884c3bd3732,4c92f76d494c8c50] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:35:21.549 INFO 2458 --- [ocp-schedules-25,0af6248ecb701990,ec2e1cb4780fb582] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:21.552 INFO 2458 --- [ocp-schedules-20,1b233884c3bd3732,4c92f76d494c8c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:21.553 INFO 2458 --- [ocp-schedules-20,1b233884c3bd3732,4c92f76d494c8c50] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:35:21.554 INFO 2458 --- [ocp-schedules-8,95b61cfd006b622a,952865f82308edc6] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:35:21.557 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:21.557 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:21.558 INFO 2458 --- [ocp-schedules-8,95b61cfd006b622a,952865f82308edc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:21.560 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:21.560 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:21.560 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:21.561 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:21.563 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:21.563 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:21.564 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:21.565 INFO 2458 --- [ocp-schedules-12,a24901c7e3692baa,8b79bc091da0058c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:35:21.566 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:21.592 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.592 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:21.593 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:35:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.616 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:21.618 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:21.619 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:21.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:35:21.622 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:21.627 INFO 2458 --- [ocp-schedules-25,0af6248ecb701990,ec2e1cb4780fb582] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:35:21.629 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:21.635 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.636 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:21.636 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:35:21.643 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:35:21.644 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:35:21.648 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:35:21.649 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:35:21.659 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:21.659 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:21.668 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:35:21.685 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:35:21.687 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:21.687 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.688 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:35:21.690 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:35:21.693 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:35:21.694 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:35:21.697 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.698 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:21.700 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:21.714 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:21.711594+08:00, lastProcessTime=2025-10-27T11:35:11.714152+08:00 2025-10-27 11:35:21.722 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:35:21.724 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:21.727 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:21.738 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:21.741 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:35:21.741 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:35:21.746 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:35:21.759 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:35:21.761 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:21.762 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:21.763 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:35:21.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.767 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:35:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:21.772 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:35:21.776 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:35:21.778 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:21.779 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.779 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:35:21.781 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:35:21.787 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:35:21.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:21.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:21.792 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:21.804 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:21.805 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:21.811 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:21.812 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:21.817 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:21.819 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:35:21.819 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:35:21.820 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:35:21.838 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:35:21.841 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:35:21.846 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:21.848 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:35:21.852 INFO 2458 --- [host-check-host-status3,59cd05ac0d8bf882,c8da34e318b1828d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:35:21.856 INFO 2458 --- [host-check-host-status5,59cd05ac0d8bf882,7638ff1fba9569ba] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:35:21.864 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:35:21.871 INFO 2458 --- [host-check-host-status2,59cd05ac0d8bf882,0d2a5770cf128e39] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:35:21.873 INFO 2458 --- [host-check-host-status4,59cd05ac0d8bf882,3337687b18554d90] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:35:21.874 INFO 2458 --- [host-check-host-status1,59cd05ac0d8bf882,110109f1a27e11db] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:35:21.874 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:21.875 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.876 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:21.878 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.878 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:21.881 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:35:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:21.914 INFO 2458 --- [ocp-schedules-29,1b050e236b18c4eb,31ae618a273864b9] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:35:21.929 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:21.929 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:21.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:35:21.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:21.970 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:21.976 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:21.711594+08:00 2025-10-27 11:35:21.991 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:35:21.991 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.992 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:35:21.994 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:21.995 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:35:22.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:22.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:22.010 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:35:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:22.033 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:22.034 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:35:22.039 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:22.040 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:22.057 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:35:22.057 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:35:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:22.111 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:22.151 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:22.153 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:22.156 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:35:22.167 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:35:22.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:22.179 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:22.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:35:22.314 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:35:22.315 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:22.315 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:35:22.318 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:22.319 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:35:22.359 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:35:22.397 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:22.398 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:22.398 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:35:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:35:22.436 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:35:22.437 INFO 2458 --- [ocp-schedules-8,95b61cfd006b622a,952865f82308edc6] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:35:22.667 WARN 2458 --- [ocp-async-3,26b80a35ab73c20d,1ab89f24c2515acc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:22.670 INFO 2458 --- [ocp-async-4,723dc625e5dda04f,e80a8e75a6280e46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:23.707 INFO 2458 --- [prometheus-minute-schedule-1,809abb636a175415,e1d8a7d07ccbbf41] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:23.714 INFO 2458 --- [prometheus-minute-schedule-1,809abb636a175415,e1d8a7d07ccbbf41] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:24.659 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:24.660 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:24.667 WARN 2458 --- [ocp-async-6,a406ab852de77bd6,9b6ee557cf1866d7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:24.670 INFO 2458 --- [ocp-async-5,7d412c850f1ea89b,6b55f9c2e763b587] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:25.870 INFO 2458 --- [ocp-schedules-26,3d06919f18dd6c4e,a43ff11fc52ad9f6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:25.873 INFO 2458 --- [ocp-schedules-26,3d06919f18dd6c4e,a43ff11fc52ad9f6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:26.668 WARN 2458 --- [ocp-async-8,ba68eaec031bde54,f82c0431275e1456] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:26.670 INFO 2458 --- [ocp-async-2,7747f9dc194d5b55,5b449341a1874879] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:26.707 INFO 2458 --- [prometheus-minute-schedule-1,1d037fe65ff783fe,11d770cd3d6847e0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:26.714 INFO 2458 --- [prometheus-minute-schedule-1,1d037fe65ff783fe,11d770cd3d6847e0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:27.142 INFO 2458 --- [ocp-schedules-3,8fbebf90d9f0ad6e,c7527b756244da8a] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:27.388 INFO 2458 --- [ocp-schedules-16,104b81541c7c0d04,8dd664111e95ebce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:27.432 WARN 2458 --- [scanner-112214845-1,104b81541c7c0d04,f2cedd95ac24164c] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:27.558 INFO 2458 --- [alarm-detect-5,104b81541c7c0d04,ca4172885784a899] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536100 ~ 1761536117 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536115, value=6.2632289776E7)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536115, value=17409.0)])]}, metricEvaluateResults={request_queue_time > 1000=[ExpressionEvaluateResult(matchValue=1000.0, metric=request_queue_time, match=true, value=6.2632289776E7, values={1761536115=6.2632289776E7}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})], ob_tenant_request_queue_size > 10=[ExpressionEvaluateResult(matchValue=10.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536115=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:35:27.558 INFO 2458 --- [alarm-detect-5,104b81541c7c0d04,ca4172885784a899] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536100 ~ 1761536117 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536115, value=6.2632289776E7)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536115, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536115=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:35:27.660 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:27.661 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:27.774 INFO 2458 --- [ocp-schedules-15,2c4e28f9abdf0747,6f1b7059809670f4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:35:27.782 INFO 2458 --- [ocp-schedules-15,2c4e28f9abdf0747,6f1b7059809670f4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:27.782 INFO 2458 --- [ocp-schedules-15,2c4e28f9abdf0747,6f1b7059809670f4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:28.542 INFO 2458 --- [ocp-async-1,dc8fe8d4f52f04fa,8922e69ecd8db1f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:28.668 WARN 2458 --- [ocp-async-7,fb561be09414bcc8,56a3d7db42f31389] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:28.670 INFO 2458 --- [ocp-async-4,4931cb16a30a9ffd,7e9638d1ebf1bc3f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:29.074 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:35:29.074 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:35:29.078 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:35:29.078 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839260767611461632 2025-10-27 11:35:29.082 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.ocp.monitor.service.IdKeyService : Sync record key and id, newRecordIdsCount=2, maxSyncedRecordId=839265989532524544 2025-10-27 11:35:29.083 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:35:29.083 INFO 2458 --- [ocp-schedules-4,77729e5b7410d28d,f06f0e49d413b386] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:35:29.522 INFO 2458 --- [ocp-schedules-19,93004245fb71eadd,db4893c7a85d066a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:35:29.526 INFO 2458 --- [ocp-schedules-19,93004245fb71eadd,db4893c7a85d066a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:35:29.526 INFO 2458 --- [ocp-schedules-19,93004245fb71eadd,db4893c7a85d066a] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:35:29.707 INFO 2458 --- [prometheus-minute-schedule-1,3f17eee3ffc41ca0,d4357f279d454865] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:29.714 INFO 2458 --- [prometheus-minute-schedule-1,3f17eee3ffc41ca0,d4357f279d454865] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:30.661 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:30.662 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:30.668 WARN 2458 --- [ocp-async-6,dab550e304b9cb7a,4adfabbff340b6cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:30.670 INFO 2458 --- [ocp-async-5,9a3dd8b32562b988,c8ad6622d533d492] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:31.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:31.714699+08:00, lastProcessTime=2025-10-27T11:35:21.711594+08:00 2025-10-27 11:35:31.920 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:31.714699+08:00 2025-10-27 11:35:32.668 WARN 2458 --- [ocp-async-8,7befef77dce658b0,1cde4c439d65f374] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:32.670 INFO 2458 --- [ocp-async-1,dba0c5144cbaae7d,c90c590ac0418a51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:32.707 INFO 2458 --- [prometheus-minute-schedule-1,53116d6bead39260,f94b900c24ea0e0b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:32.714 INFO 2458 --- [prometheus-minute-schedule-1,53116d6bead39260,f94b900c24ea0e0b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:33.662 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:33.663 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:33.989 INFO 2458 --- [ocp-schedules-7,386f70ebd0f73075,ae01958bb245e7c3] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:35:33.989 INFO 2458 --- [ocp-schedules-7,386f70ebd0f73075,ae01958bb245e7c3] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:35:34.668 WARN 2458 --- [ocp-async-10,7c5cf5ba7c83cf26,775ed03aa8e59b9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:34.671 INFO 2458 --- [ocp-async-3,c4ebde64b4a849d5,494d1cf70e986c8c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:35.707 INFO 2458 --- [prometheus-minute-schedule-1,f80222fc2ab8a4c3,ec2139f955404027] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:35.714 INFO 2458 --- [prometheus-minute-schedule-1,f80222fc2ab8a4c3,ec2139f955404027] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:35.883 INFO 2458 --- [ocp-schedules-28,6ffdeb4c584c349a,2669685566477762] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:35.887 INFO 2458 --- [ocp-schedules-28,6ffdeb4c584c349a,2669685566477762] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:36.663 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:36.664 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:36.668 WARN 2458 --- [ocp-async-4,532be073ee528609,af84857d987c8c42] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:36.671 INFO 2458 --- [ocp-async-9,f08afa805a6ec798,468f667f02b72eef] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:37.148 INFO 2458 --- [ocp-schedules-4,c553ebf5153171cb,1be061ca963973db] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:37.564 INFO 2458 --- [ocp-schedules-8,dc57b81b3449a246,cdcb2e519f3db0f6] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:37.602 WARN 2458 --- [scanner-112215101-1,dc57b81b3449a246,f6c5c59a18ed0144] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:37.708 INFO 2458 --- [prometheus-cache-monitor1,f0599cbb39a81d74,b9bb37438463d5d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20314,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:35:37.708 INFO 2458 --- [prometheus-cache-monitor1,f0599cbb39a81d74,b9bb37438463d5d5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:35:38.542 INFO 2458 --- [ocp-async-6,941935642fd36023,84c43c2f30d0d010] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:38.669 WARN 2458 --- [ocp-async-2,58634f1bb8a773c2,16a2b6051914ee2c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:38.671 INFO 2458 --- [ocp-async-1,f8a99c813ee4ce3e,17cf613df50bbad0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:38.707 INFO 2458 --- [prometheus-minute-schedule-1,69277e85113fbf40,7d1b6e576ee1f29a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:38.714 INFO 2458 --- [prometheus-minute-schedule-1,69277e85113fbf40,7d1b6e576ee1f29a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:38.849 INFO 2458 --- [ocp-schedules-28,0332ea9956d9eab7,5108dc8f183748fc] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:35:38.856 INFO 2458 --- [ocp-schedules-28,0332ea9956d9eab7,5108dc8f183748fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:38.856 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:35:38.856 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:35:38.857 INFO 2458 --- [ocp-schedules-28,0332ea9956d9eab7,5108dc8f183748fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:35:39.664 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:39.665 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:40.008 INFO 2458 --- [ocp-schedules-21,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:35:40.008 INFO 2458 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:35:40.043 INFO 2458 --- [Check obproxy1,5f6ceb29e37b7f9c,50c09767e8f4d2c4] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:35:40.043 INFO 2458 --- [Check obproxy1,5f6ceb29e37b7f9c,50c09767e8f4d2c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:35:40.046 INFO 2458 --- [Collect obproxy1,00104ef61ecfce40,3b3fcab8cda09a14] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:35:40.046 INFO 2458 --- [Collect obproxy1,00104ef61ecfce40,3b3fcab8cda09a14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:35:40.074 INFO 2458 --- [Check obproxy1,5f6ceb29e37b7f9c,50c09767e8f4d2c4] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:35:40.077 INFO 2458 --- [Collect obproxy1,00104ef61ecfce40,3b3fcab8cda09a14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:35:40.077 INFO 2458 --- [Check obproxy1,5f6ceb29e37b7f9c,50c09767e8f4d2c4] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:35:40.077 INFO 2458 --- [Check obproxy1,5f6ceb29e37b7f9c,50c09767e8f4d2c4] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:35:40.079 INFO 2458 --- [Collect obproxy1,00104ef61ecfce40,3b3fcab8cda09a14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:35:40.082 INFO 2458 --- [Collect obproxy1,00104ef61ecfce40,3b3fcab8cda09a14] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:35:40.156 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:40.156 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:35:40.156 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:35:40.156 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:35:40.156 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:35:40.156 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:35:40.166 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:40.169 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:40.169 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:40.169 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:40.170 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:40.171 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:40.172 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:40.173 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:40.173 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:40.173 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:40.222 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:35:40.222 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:35:40.224 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:35:40.226 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:35:40.226 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:35:40.226 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:35:40.226 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:35:40.226 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:35:40.227 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:35:40.228 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:35:40.230 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:35:40.230 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:35:40.230 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:35:40.230 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:35:40.234 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:35:40.235 INFO 2458 --- [ocp-schedules-24,3d986ebc7ab8fac8,3a30fd426b8f9d21] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:35:40.253 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:35:40.258 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:35:40.258 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:35:40.258 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:35:40.258 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:35:40.258 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:35:40.264 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:35:40.553 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:35:40.553 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:35:40.652 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:35:40.653 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:40.654 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:35:40.666 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:40.667 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:35:40.669 WARN 2458 --- [ocp-async-7,16aaafdd912bc33a,36642b24c96af216] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:40.671 INFO 2458 --- [ocp-async-3,d3692e5616f2601d,80e8b55bf76eeaf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:40.730 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:40.731 INFO 2458 --- [ocp-schedules-5,e3edc4593723f1b2,774754a6bee63f90] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:35:41.051 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.821234389S 2025-10-27 11:35:41.051 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:35:41.140 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:35:41.140 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:35:41.174 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:35:41.216 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:35:41.216 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:35:41.216 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:35:41.216 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:35:41.216 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:35:41.217 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:35:41.224 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:35:41.293 INFO 2458 --- [ocp-schedules-15,eaa0e8f222a46a1c,dbd97dd3548e69c3] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:35:41.338 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] o.s.boot.SpringApplication : Started application in 1.111 seconds (JVM running for 258609.764) 2025-10-27 11:35:41.338 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.112009959S 2025-10-27 11:35:41.660 INFO 2458 --- [ocp-schedules-17,604eccb79f34ea5f,28fac7caa130bf4b] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:35:41.707 INFO 2458 --- [prometheus-minute-schedule-1,e7618269bb618fab,f349dc8f32ecdb88] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:41.713 INFO 2458 --- [prometheus-minute-schedule-1,e7618269bb618fab,f349dc8f32ecdb88] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:41.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:41.721207+08:00, lastProcessTime=2025-10-27T11:35:31.714699+08:00 2025-10-27 11:35:41.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:41.721207+08:00 2025-10-27 11:35:42.665 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:42.666 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:42.669 WARN 2458 --- [ocp-async-3,2de230f7567a943d,cb50d710967c77c4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:42.671 INFO 2458 --- [ocp-async-9,64898b810cd7ce7b,89a56c6e27b1d772] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:44.303 INFO 2458 --- [ocp-schedules-13,63b64fc49f7d1105,8cb7e3b8a1070428] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:35:44.669 WARN 2458 --- [ocp-async-5,70b21b575af52cd3,6affa3c5340a7cfe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:44.671 INFO 2458 --- [ocp-async-2,bfcb651e91bcdd24,9a0ef722aa821a32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:44.707 INFO 2458 --- [prometheus-minute-schedule-1,d2c58e0315910a10,c85d21c50388e7ef] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:44.715 INFO 2458 --- [prometheus-minute-schedule-1,d2c58e0315910a10,c85d21c50388e7ef] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:45.102 INFO 2458 --- [prometheus-second-schedule-1,bf5786b308751c4f,0c4aa551eeb6944c] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:35:45.666 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:45.666 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:45.896 INFO 2458 --- [ocp-schedules-27,5ea063f362e4097e,0351bf4fbe08f6e7] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:45.899 INFO 2458 --- [ocp-schedules-27,5ea063f362e4097e,0351bf4fbe08f6e7] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:46.669 WARN 2458 --- [ocp-async-1,e73c2796e0305370,abef60fcb25c9836] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:46.671 INFO 2458 --- [ocp-async-7,c1383f4bc2a1ae6d,89e16619b10acd6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:47.155 INFO 2458 --- [ocp-schedules-20,bb00df471968886a,cc2053557b303b5b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:47.707 INFO 2458 --- [prometheus-minute-schedule-1,941f88a90d6c3646,9047b2a3d4a0774d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:47.714 INFO 2458 --- [prometheus-minute-schedule-1,941f88a90d6c3646,9047b2a3d4a0774d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:47.743 INFO 2458 --- [ocp-schedules-29,487818c3eb0706f9,f6f5a9ab8d9c9a26] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:47.779 WARN 2458 --- [scanner-112215264-1,487818c3eb0706f9,0bdff4414d1768a5] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:47.788 INFO 2458 --- [ocp-schedules-11,c6e94b816225bc97,daefc4dee08a7a94] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:35:47.795 INFO 2458 --- [ocp-schedules-11,c6e94b816225bc97,daefc4dee08a7a94] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:47.795 INFO 2458 --- [ocp-schedules-11,c6e94b816225bc97,daefc4dee08a7a94] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:35:47.871 INFO 2458 --- [alarm-detect-17,487818c3eb0706f9,5d48f7d0eb62306d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536077 ~ 1761536137 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=204434.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204434.56, values={1761536100=204434.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:35:47.913 INFO 2458 --- [alarm-detect-14,487818c3eb0706f9,a367c9041d647994] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536077 ~ 1761536137 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=93.233)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.233, values={1761536100=93.233}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:35:47.930 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,eb77d516caaef50b,772d4255ee0fe11b] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=eb77d516caaef50b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:35:47.930 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,eb77d516caaef50b,772d4255ee0fe11b] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=eb77d516caaef50b, duration=2 ms] 2025-10-27 11:35:47.936 INFO 2458 --- [alarm-detect-18,487818c3eb0706f9,1434340ea0fe3ac6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536077 ~ 1761536137 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536100, value=204435.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204435.912, values={1761536100=204435.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:35:47.940 INFO 2458 --- [alarm-detect-7,487818c3eb0706f9,8dfd0bae1ec11e69] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536115 ~ 1761536137 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536130, value=50.393)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536130, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536130=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:35:47.942 INFO 2458 --- [alarm-detect-7,487818c3eb0706f9,8dfd0bae1ec11e69] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:35:47.942 INFO 2458 --- [alarm-detect-7,487818c3eb0706f9,8dfd0bae1ec11e69] c.o.o.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=ob_tenant_request_queue_over_threshold, target=alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-2, activeAt=2025-10-27T11:35:17.362+08:00 2025-10-27 11:35:48.542 INFO 2458 --- [ocp-async-2,26358dcbe8157224,4ef09eca5876ee6b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:48.667 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:48.668 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:48.669 WARN 2458 --- [ocp-async-4,c403d631a67e813e,14efa3da1cdc97b1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:48.672 INFO 2458 --- [ocp-async-3,5f807158dd92dd80,0b09d6a393e20a53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:50.016 INFO 2458 --- [ocp-schedules-18,b8647afcd3e82181,d59ac7fb3668d464] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:35:50.016+08:00, clusters=[1, 2] 2025-10-27 11:35:50.051 INFO 2458 --- [ocp-schedules-18,b8647afcd3e82181,d59ac7fb3668d464] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:35:50.053 INFO 2458 --- [ocp-schedules-18,b8647afcd3e82181,d59ac7fb3668d464] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:35:50.669 WARN 2458 --- [ocp-async-8,f6c808ff54d407b5,9a494b6bacd89f0d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:50.672 INFO 2458 --- [ocp-async-9,512f91c840903a83,33f7c4c36cc394a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:50.707 INFO 2458 --- [prometheus-minute-schedule-1,c8213da396a10264,52d9a1be1aae868e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:50.714 INFO 2458 --- [prometheus-minute-schedule-1,c8213da396a10264,52d9a1be1aae868e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:51.409 INFO 2458 --- [ocp-schedules-5,d263ed09a313c048,0e31d12a7e9a08b1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:51.409 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:35:51.409 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:35:51.409 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:35:51.409 INFO 2458 --- [ocp-schedules-18,ed913958cff37074,ac53a4f4228a19c4] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:35:51.410 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:35:51.423 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:51.424 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:51.424 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:51.424 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:51.424 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:51.424 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:51.425 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:51.425 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:51.428 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:51.429 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:51.439 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:51.439 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:51.439 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:51.439 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:51.440 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:51.440 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:51.441 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:51.441 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:51.442 INFO 2458 --- [ocp-schedules-10,a73f6b42d317f2fe,79c9c6cad34e8533] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:35:51.442 INFO 2458 --- [ocp-schedules-16,f42d0ecfd9218a7a,d2b23d14f53a9ac7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:35:51.443 INFO 2458 --- [host-check-clockdiff1,f42d0ecfd9218a7a,95fa2871e1c3751b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:35:51.443 INFO 2458 --- [host-check-clockdiff2,f42d0ecfd9218a7a,a34e9b44489fa39d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:35:51.443 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:51.443 INFO 2458 --- [host-check-clockdiff4,f42d0ecfd9218a7a,e9dc619f3bfc516f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:35:51.443 INFO 2458 --- [host-check-clockdiff3,f42d0ecfd9218a7a,1ba71b8e38cdcc04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:35:51.444 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:51.444 INFO 2458 --- [host-check-clockdiff5,f42d0ecfd9218a7a,f2bc2ccf51cdeb19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:35:51.457 INFO 2458 --- [host-check-clockdiff2,f42d0ecfd9218a7a,a34e9b44489fa39d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536151 -1 -1 , error= 2025-10-27 11:35:51.485 INFO 2458 --- [host-check-clockdiff4,f42d0ecfd9218a7a,e9dc619f3bfc516f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536151 0 0 , error= 2025-10-27 11:35:51.487 INFO 2458 --- [host-check-clockdiff1,f42d0ecfd9218a7a,95fa2871e1c3751b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536151 0 0 , error= 2025-10-27 11:35:51.493 INFO 2458 --- [host-check-clockdiff5,f42d0ecfd9218a7a,f2bc2ccf51cdeb19] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536151 0 0 , error= 2025-10-27 11:35:51.496 INFO 2458 --- [host-check-clockdiff3,f42d0ecfd9218a7a,1ba71b8e38cdcc04] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536151 0 0 , error= 2025-10-27 11:35:51.521 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:35:51.524 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:35:51.523153092+08:00 2025-10-27 11:35:51.524 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:35:51.524+08:00 2025-10-27 11:35:51.532 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:35:51.532 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:35:51.534 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:35:51.533776996+08:00 2025-10-27 11:35:51.534 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:35:51.534 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:35:51.534+08:00 2025-10-27 11:35:51.537 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:35:51.539 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:35:51.539 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:35:51.539 INFO 2458 --- [host-check-agent-status3,ed913958cff37074,f4599dc31501955b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:35:51.540 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:35:51.540 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:35:51.540 INFO 2458 --- [host-check-agent-status1,ed913958cff37074,2ab59549086180e4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:35:51.540 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:35:51.539667332+08:00 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:35:51.524+08:00 and status AVAILABLE 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:35:51.541573422+08:00 2025-10-27 11:35:51.541 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:35:51.541+08:00 2025-10-27 11:35:51.542 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:35:51.541+08:00 2025-10-27 11:35:51.543 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:35:51.534+08:00 and status AVAILABLE 2025-10-27 11:35:51.543 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:35:51.543327007+08:00 2025-10-27 11:35:51.544 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:35:51.543+08:00 2025-10-27 11:35:51.545 INFO 2458 --- [ocp-schedules-6,65f9323832b358c1,b2d21ea7706d55c3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:35:51.546 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:35:51.547 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:35:51.547 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:35:51.547 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:35:51.547 INFO 2458 --- [host-check-agent-status2,ed913958cff37074,40d32d6f0546f2df] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:35:51.548 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:35:51.549 INFO 2458 --- [ocp-schedules-13,831c0115a824efd4,32db7acd6dace8b4] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:35:51.549 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:35:51.549 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:35:51.549 INFO 2458 --- [host-check-agent-status5,ed913958cff37074,fbcbd51864c4bff6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:35:51.549 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:35:51.541+08:00 and status AVAILABLE 2025-10-27 11:35:51.550 INFO 2458 --- [ocp-schedules-6,65f9323832b358c1,b2d21ea7706d55c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:51.551 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:35:51.541+08:00 and status AVAILABLE 2025-10-27 11:35:51.552 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:35:51.543+08:00 and status AVAILABLE 2025-10-27 11:35:51.553 INFO 2458 --- [ocp-schedules-13,831c0115a824efd4,32db7acd6dace8b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:51.554 INFO 2458 --- [ocp-schedules-25,a8f1fdbf3912050b,301836280f1e9355] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:35:51.554 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:35:51.554 INFO 2458 --- [ocp-schedules-13,831c0115a824efd4,32db7acd6dace8b4] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:35:51.556 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:35:51.556 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:35:51.556 INFO 2458 --- [host-check-agent-status4,ed913958cff37074,482f0da22a158494] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:35:51.557 INFO 2458 --- [ocp-schedules-25,a8f1fdbf3912050b,301836280f1e9355] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:51.558 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:35:51.561 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:35:51.561 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:35:51.565 INFO 2458 --- [ocp-schedules-15,bff144ad058b3c64,54beb29e1748dfde] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:35:51.568 INFO 2458 --- [ocp-schedules-17,77ad5893dc93a3cc,06489a91e561ec04] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:35:51.569 INFO 2458 --- [ocp-schedules-4,ad425bfcdfd61de6,d5df7e895a370cc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:35:51.569 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:35:51.572 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:35:51.572 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:35:51.574 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:35:51.579 INFO 2458 --- [ocp-schedules-17,77ad5893dc93a3cc,06489a91e561ec04] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:35:51.581 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:35:51.581 INFO 2458 --- [ocp-schedules-4,ad425bfcdfd61de6,d5df7e895a370cc5] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:35:51.582 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:51.583 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:51.584 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:35:51.585 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:51.585 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:51.590 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.591 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:51.592 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:35:51.613 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.614 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:51.618 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:35:51.618 INFO 2458 --- [ocp-schedules-6,65f9323832b358c1,b2d21ea7706d55c3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:35:51.631 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.632 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:51.635 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:51.639 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:51.639 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:51.646 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:35:51.647 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:35:51.655 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:35:51.658 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:35:51.662 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:35:51.668 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:51.668 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:51.677 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:35:51.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:51.689 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:35:51.690 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:35:51.696 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:51.697 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:35:51.702 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.703 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:35:51.704 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:35:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:35:51.714409+08:00, lastProcessTime=2025-10-27T11:35:41.721207+08:00 2025-10-27 11:35:51.716 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:35:51.741 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:51.741 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:51.741 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:51.744 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:51.753 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:35:51.758 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:35:51.759 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:35:51.761 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:35:51.762 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:35:51.789 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:51.790 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.790 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:35:51.791 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:35:51.792 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:51.793 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.793 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:51.798 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.802 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:35:51.802 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:51.810 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:35:51.812 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:35:51.817 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:35:51.818 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:35:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:51.835 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:35:51.841 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:51.845 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:51.849 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:51.854 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:51.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:51.855 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:35:51.863 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:35:51.866 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:35:51.870 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:35:51.876 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:35:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:35:51.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:35:51.889 INFO 2458 --- [host-check-host-status4,d263ed09a313c048,69304c9675e4cd3a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:35:51.890 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:51.891 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:51.891 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:51.891 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:51.892 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:35:51.897 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:35:51.905 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:35:51.911 INFO 2458 --- [host-check-host-status2,d263ed09a313c048,2a96bd2fac84e8ac] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:35:51.911 INFO 2458 --- [host-check-host-status5,d263ed09a313c048,51fc0b8484103d9b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:35:51.912 INFO 2458 --- [host-check-host-status3,d263ed09a313c048,db827cc6553641f0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:35:51.916 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:35:51.917 INFO 2458 --- [ocp-schedules-30,b1634194f2910a16,1368a229e8954e4f] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:35:51.928 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:35:51.937 INFO 2458 --- [host-check-host-status1,d263ed09a313c048,f9fddc02f526fccc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:35:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:51.979 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:35:51.985 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:51.985 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:51.989 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:35:51.714409+08:00 2025-10-27 11:35:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:52.022 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:35:52.022 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.023 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:35:52.028 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.029 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:35:52.055 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:35:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:52.063 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:35:52.076 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:52.080 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.081 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:35:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:35:52.105 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:35:52.105 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:35:52.137 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:35:52.175 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:52.176 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:35:52.181 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:35:52.191 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:35:52.200 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:35:52.203 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:35:52.258 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:35:52.328 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:35:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.329 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:35:52.332 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.332 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:35:52.369 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:35:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:35:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:35:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:35:52.453 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:35:52.453 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:35:52.453 INFO 2458 --- [ocp-schedules-25,a8f1fdbf3912050b,301836280f1e9355] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:35:52.670 WARN 2458 --- [ocp-async-2,a5051bd6e1287ebf,d936d35be7a1c0cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:52.672 INFO 2458 --- [ocp-async-4,19d3c40274815659,c6839c0be7bd0f8b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:35:53.236 INFO 2458 --- [ocp-schedules-3,094086aaa3236e36,93bafbd4dd115016] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20314,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5082,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7892} 2025-10-27 11:35:53.237 INFO 2458 --- [ocp-schedules-3,094086aaa3236e36,93bafbd4dd115016] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:35:53.707 INFO 2458 --- [prometheus-minute-schedule-1,60b595cebbc3d087,f49407288a799bd6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:53.714 INFO 2458 --- [prometheus-minute-schedule-1,60b595cebbc3d087,f49407288a799bd6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:54.668 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:54.669 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:54.670 WARN 2458 --- [ocp-async-4,8c8eca574a5bb15f,f5ac59126594f330] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:54.672 INFO 2458 --- [ocp-async-8,b8dea8dedaa867f8,f4bce1c960c7dd19] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:55.755 INFO 2458 --- [ocp-schedules-24,8cbbd60fe6e95976,70997fe6b59535e8] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:35:55.910 INFO 2458 --- [ocp-schedules-10,4fa7e1fe668ff57f,cdd3edb092fcee30] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:35:55.913 INFO 2458 --- [ocp-schedules-10,4fa7e1fe668ff57f,cdd3edb092fcee30] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:35:56.670 WARN 2458 --- [ocp-async-8,0a923c7e62a3a983,8a7adfb50df3d576] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:56.672 INFO 2458 --- [ocp-async-1,341cc8022c3af002,4e30c10c0e7c5ff4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:56.707 INFO 2458 --- [prometheus-minute-schedule-1,d6245859cff70d2d,4309f7c7003a9749] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:56.714 INFO 2458 --- [prometheus-minute-schedule-1,d6245859cff70d2d,4309f7c7003a9749] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:35:57.162 INFO 2458 --- [ocp-schedules-8,ba14ce8e1df47149,915e0f0b4835ab0d] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:35:57.669 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:35:57.670 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:35:57.979 INFO 2458 --- [ocp-schedules-10,a18568b00fb16851,bc8022615b25f153] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:35:58.001 WARN 2458 --- [scanner-112215533-1,a18568b00fb16851,6c2016f2aac91780] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:35:58.089 INFO 2458 --- [alarm-detect-5,a18568b00fb16851,7c656cb193eb6904] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536130 ~ 1761536148 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536145, value=48.704)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536145, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536145=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:35:58.453 INFO 2458 --- [ocp-schedules-8,1c3e3006bc783fe8,4c652a327e856817] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:35:58.459 INFO 2458 --- [ocp-schedules-8,1c3e3006bc783fe8,4c652a327e856817] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:35:58.505 INFO 2458 --- [ocp-schedules-8,1c3e3006bc783fe8,4c652a327e856817] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:35:58.542 INFO 2458 --- [ocp-async-1,36fa428ebe5d7993,619dea6d78422771] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:35:58.670 WARN 2458 --- [ocp-async-2,e29bdb20acd9cf66,d16fa2ae53f41d76] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:35:58.672 INFO 2458 --- [ocp-async-6,c15f686b938c5726,7385e062c32d412a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:35:59.580 INFO 2458 --- [ocp-schedules-30,cb4c7ab02a49a4cb,a206dd4372b1a75c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:35:59.707 INFO 2458 --- [prometheus-minute-schedule-1,c16ec6188515a12e,772b37d6b0921662] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536160, interval=60 2025-10-27 11:35:59.715 INFO 2458 --- [prometheus-minute-schedule-1,c16ec6188515a12e,772b37d6b0921662] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:00.670 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:00.670 WARN 2458 --- [ocp-async-6,6a13d7231e8087d9,86e4cd54007e0f43] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:00.671 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:00.672 INFO 2458 --- [ocp-async-3,db7285abb16a64fd,546038c61a34f2f7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:01.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:01.718347+08:00, lastProcessTime=2025-10-27T11:35:51.714409+08:00 2025-10-27 11:36:01.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:01.718347+08:00 2025-10-27 11:36:02.670 WARN 2458 --- [ocp-async-8,d931083d9173e1c1,60e2b52b55c83aa1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:02.673 INFO 2458 --- [ocp-async-1,8192fa8207532cdc,2d7e57b814f594d1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:02.707 INFO 2458 --- [prometheus-minute-schedule-1,3051ffbec134e4b6,b12fbd7b905de17e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:02.714 INFO 2458 --- [prometheus-minute-schedule-1,3051ffbec134e4b6,b12fbd7b905de17e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:03.671 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:03.672 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:04.671 WARN 2458 --- [ocp-async-2,259abf6a1f8e1f74,045355769a8ca88f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:04.673 INFO 2458 --- [ocp-async-4,36df6e69a3e2fcc4,33927c968ec798b9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:04.783 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,4e78e9f78c8642f0,4903806caca26900] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=4e78e9f78c8642f0, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:36:04.783 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,4e78e9f78c8642f0,4903806caca26900] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=4e78e9f78c8642f0, duration=1 ms] 2025-10-27 11:36:04.909 INFO 2458 --- [ocp-schedules-11,9f45d9e44e33d7bd,3abc2a849061fe28] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:36:04.917 INFO 2458 --- [ocp-schedules-11,9f45d9e44e33d7bd,3abc2a849061fe28] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:04.951 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:04.951 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:36:04.966 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:04.967 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:36:04.983 INFO 2458 --- [ocp-schedules-11,9f45d9e44e33d7bd,3abc2a849061fe28] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:36:05.707 INFO 2458 --- [prometheus-minute-schedule-1,6ea32b3b61df49c8,4a1965b57aed9116] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:05.714 INFO 2458 --- [prometheus-minute-schedule-1,6ea32b3b61df49c8,4a1965b57aed9116] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:05.920 INFO 2458 --- [ocp-schedules-22,aefceede04f388fd,61d9d0d9a007e6cd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:05.924 INFO 2458 --- [ocp-schedules-22,aefceede04f388fd,61d9d0d9a007e6cd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:06.023 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:06.024 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:36:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:06.026 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:36:06.028 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:36:06.078 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:36:06.116 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:36:06.141 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:36:06.141 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:36:06.141 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:36:06.141 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:36:06.143 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:36:06.144 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:06.145 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:36:06.147 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:06.147 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:36:06.411 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:33:30, start=2025-10-27T11:30:00, end=2025-10-27T11:32:30, updates=208, granule=120000ms, delay=210000ms 2025-10-27 11:36:06.422 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:33:30, target=2025-10-27T11:32:30, granule=120000ms, delay=60000ms) 2025-10-27 11:36:06.422 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:32:30, target=2025-10-27T11:22:30, granule=600000ms, delay=600000ms) 2025-10-27 11:36:06.422 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=394ms, period=120000ms 2025-10-27 11:36:06.671 WARN 2458 --- [ocp-async-5,f5b04c9e05b66628,0efd60142bcd7b3c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:06.672 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:06.673 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:06.673 INFO 2458 --- [ocp-async-3,9f4892a0bee8a1d0,676a644cbd209a1f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:07.168 INFO 2458 --- [ocp-schedules-23,88a09d8f7b86ca53,410e036913f87631] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:07.798 INFO 2458 --- [ocp-schedules-16,423417838fac3dd1,ed51a67017c449ed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:36:07.806 INFO 2458 --- [ocp-schedules-16,423417838fac3dd1,ed51a67017c449ed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:07.806 INFO 2458 --- [ocp-schedules-16,423417838fac3dd1,ed51a67017c449ed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:08.095 INFO 2458 --- [ocp-schedules-28,3803edef0e41085d,6020e5036f3ece3d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:08.150 WARN 2458 --- [scanner-112215720-1,3803edef0e41085d,bca28defd738e55a] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:08.409 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:36:08.409+08:00, tenantSize=6 2025-10-27 11:36:08.436 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761536168409000 2025-10-27 11:36:08.436 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761535868262000 2025-10-27 11:36:08.497 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:36:08.507 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761536168501000 2025-10-27 11:36:08.507 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761535868376000 2025-10-27 11:36:08.530 INFO 2458 --- [ocp-schedules-10,8fdb831a83faf161,c010f70f70b4a7f9] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:36:08.543 INFO 2458 --- [ocp-async-3,944151cb1ba60bb2,ba43cf8272c522b5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:08.671 WARN 2458 --- [ocp-async-9,3d79d31c4b805e00,2a8b6d73f6290ac5] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:08.673 INFO 2458 --- [ocp-async-1,afc2c6a14c4c9762,bd8a3a21c73954fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:08.707 INFO 2458 --- [prometheus-minute-schedule-1,5847a5ecdb0eb59c,0069997bb9783cdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:08.714 INFO 2458 --- [prometheus-minute-schedule-1,5847a5ecdb0eb59c,0069997bb9783cdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:09.673 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:09.674 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:10.009 INFO 2458 --- [ocp-schedules-24,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:36:10.010 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:36:10.043 INFO 2458 --- [Check obproxy1,731fc1911a31cfa3,6051b10f8cd86a61] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:36:10.046 INFO 2458 --- [Check obproxy1,731fc1911a31cfa3,6051b10f8cd86a61] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:36:10.053 INFO 2458 --- [Collect obproxy1,91db30dd35e03190,78716e3429694175] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:36:10.053 INFO 2458 --- [Collect obproxy1,91db30dd35e03190,78716e3429694175] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:36:10.076 INFO 2458 --- [Check obproxy1,731fc1911a31cfa3,6051b10f8cd86a61] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:36:10.080 INFO 2458 --- [Check obproxy1,731fc1911a31cfa3,6051b10f8cd86a61] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:36:10.080 INFO 2458 --- [Check obproxy1,731fc1911a31cfa3,6051b10f8cd86a61] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:36:10.084 INFO 2458 --- [Collect obproxy1,91db30dd35e03190,78716e3429694175] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:36:10.088 INFO 2458 --- [Collect obproxy1,91db30dd35e03190,78716e3429694175] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:36:10.090 INFO 2458 --- [Collect obproxy1,91db30dd35e03190,78716e3429694175] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:36:10.185 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:36:10.671 WARN 2458 --- [ocp-async-2,e84e715e5c587dbc,b6b78bea3aadc8e6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:10.673 INFO 2458 --- [ocp-async-4,e5278f53aa52c5b5,f15462aa5c09b7fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:11.707 INFO 2458 --- [prometheus-minute-schedule-1,f0e0deedf112d5cf,bd6660684f8b1eae] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:11.713 INFO 2458 --- [prometheus-minute-schedule-1,f0e0deedf112d5cf,bd6660684f8b1eae] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:11.714715+08:00, lastProcessTime=2025-10-27T11:36:01.718347+08:00 2025-10-27 11:36:11.779 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=1000, affectRows=1000 2025-10-27 11:36:11.784 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=994, validCount=992 2025-10-27 11:36:11.883 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=1000, affectRows=1000 2025-10-27 11:36:11.886 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=999, validCount=995 2025-10-27 11:36:11.905 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570103) Duplicate entry '835014129065861120-1761536040' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570103) Duplicate entry '835014129065861120-1761536040' for key 'PRIMARY' 2025-10-27 11:36:11.929 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:11.714715+08:00 2025-10-27 11:36:12.150 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=1000, affectRows=1000 2025-10-27 11:36:12.155 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=992, validCount=991 2025-10-27 11:36:12.272 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=1000, affectRows=1000 2025-10-27 11:36:12.276 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=1000, validCount=993 2025-10-27 11:36:12.299 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=82, affectRows=82 2025-10-27 11:36:12.299 INFO 2458 --- [metric-archive-second-5,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=82, validCount=82 2025-10-27 11:36:12.671 WARN 2458 --- [ocp-async-5,5a51c5442a01237f,f1ae64e2ed3f3e1a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:12.673 INFO 2458 --- [ocp-async-3,caed73ce99cf6bbd,c73d7ab153e5c1ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:12.674 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:12.674 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:12.898 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536100, nodeCount=817, affectRows=817 2025-10-27 11:36:14.671 WARN 2458 --- [ocp-async-9,e6f843d4b0f081a4,29c08022efc37891] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:14.673 INFO 2458 --- [ocp-async-10,6af6d447fb8521f2,b214f8fdc279d5f9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:14.707 INFO 2458 --- [prometheus-minute-schedule-1,2f1a9295eedc4ab5,8c8597b3a5043a47] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:14.714 INFO 2458 --- [prometheus-minute-schedule-1,2f1a9295eedc4ab5,8c8597b3a5043a47] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:15.675 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:15.675 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:15.931 INFO 2458 --- [ocp-schedules-26,c5340aa71bdfadf7,582d71ce221f21ae] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:15.934 INFO 2458 --- [ocp-schedules-26,c5340aa71bdfadf7,582d71ce221f21ae] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:16.029 INFO 2458 --- [ocp-schedules-16,feecf27e18448ba4,32412fbfb42a53c5] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:36:16.036 INFO 2458 --- [ocp-schedules-16,feecf27e18448ba4,32412fbfb42a53c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:16.132 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:36:16.132 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.135 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:36:16.147 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.147 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.173 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.174 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.206 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.207 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.209 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.209 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.223 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:36:16.223 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.228 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:36:16.238 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.239 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.249 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.250 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.264 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.264 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.283 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.284 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.297 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.298 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.299 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.300 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.301 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.301 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.302 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.303 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.304 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.304 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.305 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.306 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.306 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:36:16.313 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:36:16.314 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.315 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:36:16.318 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.318 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.348 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.348 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.371 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:16.372 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.372 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:36:16.373 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.373 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:36:16.376 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.376 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:36:16.383 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.383 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:36:16.383 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.383 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:36:16.386 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:36:16.386 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:36:16.409 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.410 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:36:16.439 INFO 2458 --- [ocp-schedules-7,359cc82a61908c2f,be6579ecb1a8b2b4] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:36:16.439+08:00, clusters=[1, 2] 2025-10-27 11:36:16.441 INFO 2458 --- [ocp-schedules-7,359cc82a61908c2f,be6579ecb1a8b2b4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:36:16.441 INFO 2458 --- [ocp-schedules-7,359cc82a61908c2f,be6579ecb1a8b2b4] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:36:16.447 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.448 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:36:16.451 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.451 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:36:16.482 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.483 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:36:16.519 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.520 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:36:16.569 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.570 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.571 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.572 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.573 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.574 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.575 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.575 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.576 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.577 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.577 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.578 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.579 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.580 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.580 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.581 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.582 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.582 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.583 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:36:16.584 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.585 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:36:16.585 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.586 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:36:16.587 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:36:16.590 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:36:16.671 WARN 2458 --- [ocp-async-7,aa558a0291ecf186,7f702cb3ef6823e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:16.674 INFO 2458 --- [ocp-async-2,23aee15cfc550b8d,c969cd4f7457d604] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:16.693 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:16.694 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:16.694 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:36:16.721 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:36:16.725 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:36:16.725 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:36:16.725 INFO 2458 --- [ocp-schedules-16,feecf27e18448ba4,32412fbfb42a53c5] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:36:16.744 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536100, nodeCount=1000, affectRows=1000 2025-10-27 11:36:16.748 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536100, nodeCount=999, validCount=986 2025-10-27 11:36:17.175 INFO 2458 --- [ocp-schedules-22,e23d2df5e3285b0f,69eb74ff0de02cba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:17.707 INFO 2458 --- [prometheus-minute-schedule-1,d8477cab3adb6be1,8f9c0367f5fe5d14] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:17.714 INFO 2458 --- [prometheus-minute-schedule-1,d8477cab3adb6be1,8f9c0367f5fe5d14] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:18.287 INFO 2458 --- [ocp-schedules-9,5e0843156d58ec32,0161e938f4f9a6e2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:18.299 WARN 2458 --- [scanner-112217503-1,5e0843156d58ec32,fbdb93eea3c98720] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:18.420 INFO 2458 --- [alarm-detect-6,5e0843156d58ec32,58108435ad718d16] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536145 ~ 1761536168 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536160, value=50.338)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536160, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536160=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:36:18.459 INFO 2458 --- [ocp-schedules-11,0d17b0f35e61c2d2,6c6ed0e946a5def9] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:36:18.459 INFO 2458 --- [ocp-schedules-11,0d17b0f35e61c2d2,6c6ed0e946a5def9] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:36:18.513 INFO 2458 --- [Collect all tenants major compactions2,0d17b0f35e61c2d2,7b6607ddb0f67057] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:36:18.515 INFO 2458 --- [Collect all tenants major compactions1,0d17b0f35e61c2d2,e737e0a0491d273b] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:36:18.517 INFO 2458 --- [Collect all tenants major compactions3,0d17b0f35e61c2d2,7401ebc7f8a57bc5] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:36:18.521 INFO 2458 --- [Collect all tenants major compactions4,0d17b0f35e61c2d2,e7ce510a0e5f92a0] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:36:18.524 INFO 2458 --- [Collect all tenants major compactions5,0d17b0f35e61c2d2,c5f44c7aeb3fa3ad] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:36:18.535 INFO 2458 --- [Collect all tenants major compactions6,0d17b0f35e61c2d2,53545020d46440dd] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:36:18.543 INFO 2458 --- [ocp-async-2,2bb6a7adec172c61,ab8548678063096b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:18.561 INFO 2458 --- [Collect all tenants major compactions5,0d17b0f35e61c2d2,c5f44c7aeb3fa3ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.567 INFO 2458 --- [Collect all tenants major compactions5,0d17b0f35e61c2d2,c5f44c7aeb3fa3ad] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:36:18.570 INFO 2458 --- [Collect all tenants major compactions2,0d17b0f35e61c2d2,7b6607ddb0f67057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.571 INFO 2458 --- [Collect all tenants major compactions2,0d17b0f35e61c2d2,7b6607ddb0f67057] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:36:18.572 INFO 2458 --- [Collect all tenants major compactions6,0d17b0f35e61c2d2,53545020d46440dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.574 INFO 2458 --- [Collect all tenants major compactions6,0d17b0f35e61c2d2,53545020d46440dd] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:36:18.574 INFO 2458 --- [Collect all tenants major compactions4,0d17b0f35e61c2d2,e7ce510a0e5f92a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.576 INFO 2458 --- [Collect all tenants major compactions5,0d17b0f35e61c2d2,c5f44c7aeb3fa3ad] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:36:18.577 INFO 2458 --- [Collect all tenants major compactions6,0d17b0f35e61c2d2,53545020d46440dd] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:36:18.578 INFO 2458 --- [Collect all tenants major compactions3,0d17b0f35e61c2d2,7401ebc7f8a57bc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.578 INFO 2458 --- [Collect all tenants major compactions2,0d17b0f35e61c2d2,7b6607ddb0f67057] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:36:18.579 INFO 2458 --- [Collect all tenants major compactions3,0d17b0f35e61c2d2,7401ebc7f8a57bc5] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:36:18.579 INFO 2458 --- [Collect all tenants major compactions4,0d17b0f35e61c2d2,e7ce510a0e5f92a0] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:36:18.580 INFO 2458 --- [Collect all tenants major compactions1,0d17b0f35e61c2d2,e737e0a0491d273b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:18.581 INFO 2458 --- [Collect all tenants major compactions1,0d17b0f35e61c2d2,e737e0a0491d273b] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:36:18.583 INFO 2458 --- [Collect all tenants major compactions4,0d17b0f35e61c2d2,e7ce510a0e5f92a0] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:36:18.583 INFO 2458 --- [Collect all tenants major compactions3,0d17b0f35e61c2d2,7401ebc7f8a57bc5] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:36:18.585 INFO 2458 --- [Collect all tenants major compactions1,0d17b0f35e61c2d2,e737e0a0491d273b] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:36:18.586 INFO 2458 --- [ocp-schedules-11,0d17b0f35e61c2d2,6c6ed0e946a5def9] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:36:18.672 WARN 2458 --- [ocp-async-6,d73f21f8f60ff3ed,ec929e569713cc96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:18.674 INFO 2458 --- [ocp-async-3,758cb9069c0c0845,31eb226843366330] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:18.675 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:18.676 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:20.198 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_INDEX_DIAGNOSER, Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:20.207 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterV 2025-10-27 11:36:20.209 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:20.210 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:20.212 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:20.672 WARN 2458 --- [ocp-async-8,e3e0d57d792e1318,d029edf90364270f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:20.674 INFO 2458 --- [ocp-async-10,52b4efc725bd85f1,787b56ea07e596ce] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:20.707 INFO 2458 --- [prometheus-minute-schedule-1,4fdd136b4ea7f1c5,6d53dbb5efbb1de7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:20.713 INFO 2458 --- [prometheus-minute-schedule-1,4fdd136b4ea7f1c5,6d53dbb5efbb1de7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:20.814 INFO 2458 --- [ocp-schedules-10,1d028d737c17dffb,6d059927c7be01c3] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:36:21.218 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.231 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.237 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.245 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.254 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.255 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.256 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.257 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.259 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.265 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.268 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.272 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.277 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.278 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.279 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.280 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.281 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.283 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : INEFFECTIVE_HINT found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.284 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.285 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : TABLE_SCAN_INDEX_NOT_EXISTS found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.286 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.SqlIndexDiagnoseService : AWFUL_PERFORMANCE_INDEX_USED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:30:18.931, end=2025-10-27T11:35:20.193} 2025-10-27 11:36:21.286 INFO 2458 --- [pool-18-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_INDEX_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:36:21.409 INFO 2458 --- [ocp-schedules-22,c8c0adb1a2f6e8bb,c3ad8d18b34e7e80] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:21.409 INFO 2458 --- [ocp-schedules-1,b4f5f0912b86ad8e,bf7baa0dde8774fa] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:21.409 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:36:21.409 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:36:21.411 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:36:21.411 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:36:21.412 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:36:21.413 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:36:21.414 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:36:21.414 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:36:21.414 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:36:21.414 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:36:21.426 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:21.426 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:21.427 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:21.427 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:21.428 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:21.428 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:21.429 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:21.431 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:21.431 INFO 2458 --- [ocp-schedules-26,0c334b299bf01cbf,1c9020a83ac69e9c] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:21.431 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:21.431 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:21.431 INFO 2458 --- [host-check-clockdiff1,0c334b299bf01cbf,2ed3c45203b42401] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:36:21.432 INFO 2458 --- [host-check-clockdiff2,0c334b299bf01cbf,4b6247c948be6881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:36:21.432 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:21.432 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:21.433 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:21.433 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:21.433 INFO 2458 --- [host-check-clockdiff3,0c334b299bf01cbf,547f4029bad56f2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:36:21.433 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:21.433 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:21.434 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:21.434 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:21.435 INFO 2458 --- [host-check-clockdiff4,0c334b299bf01cbf,6cbe789de26e121a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:36:21.440 INFO 2458 --- [host-check-clockdiff5,0c334b299bf01cbf,5832e3488a1de1b0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:36:21.445 INFO 2458 --- [ocp-schedules-29,e87097e61d5685c7,24ba6745070f707c] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:36:21.447 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:21.448 INFO 2458 --- [host-check-clockdiff2,0c334b299bf01cbf,4b6247c948be6881] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536181 -1 -1 , error= 2025-10-27 11:36:21.451 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:21.459 INFO 2458 --- [host-check-clockdiff1,0c334b299bf01cbf,2ed3c45203b42401] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536181 0 0 , error= 2025-10-27 11:36:21.466 INFO 2458 --- [host-check-clockdiff5,0c334b299bf01cbf,5832e3488a1de1b0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536181 0 0 , error= 2025-10-27 11:36:21.510 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:36:21.510 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:36:21.511 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:36:21.511577464+08:00 2025-10-27 11:36:21.512 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:36:21.513 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:36:21.513 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:36:21.512+08:00 2025-10-27 11:36:21.513 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:36:21.512959323+08:00 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-agent-status5,c8c0adb1a2f6e8bb,92331e4fe8802ac7] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:36:21.514355409+08:00 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:36:21.513438976+08:00 2025-10-27 11:36:21.514 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:36:21.513+08:00 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:36:21.514+08:00 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status4,c8c0adb1a2f6e8bb,73e7a5a73eb5c657] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-agent-status3,c8c0adb1a2f6e8bb,4f674c344d402421] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:36:21.515 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:36:21.514+08:00 2025-10-27 11:36:21.516 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:36:21.515730048+08:00 2025-10-27 11:36:21.516 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:36:21.516 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:36:21.516 INFO 2458 --- [host-check-agent-status2,c8c0adb1a2f6e8bb,c95927107cdd7747] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:36:21.516 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:36:21.516+08:00 2025-10-27 11:36:21.520 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:36:21.512+08:00 and status AVAILABLE 2025-10-27 11:36:21.522 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:36:21.523 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:36:21.514+08:00 and status AVAILABLE 2025-10-27 11:36:21.523 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:36:21.513+08:00 and status AVAILABLE 2025-10-27 11:36:21.524 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:36:21.516+08:00 and status AVAILABLE 2025-10-27 11:36:21.525 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:36:21.514+08:00 and status AVAILABLE 2025-10-27 11:36:21.525 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:36:21.526 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:36:21.526 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:36:21.526 INFO 2458 --- [host-check-agent-status1,c8c0adb1a2f6e8bb,5e6a4325ebbd767c] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:36:21.527 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:36:21.529 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:36:21.529 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:36:21.529 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:36:21.545 INFO 2458 --- [ocp-schedules-20,9c88958dc07fa3b8,c0f894a8c64a82d3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:36:21.546 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:21.549 INFO 2458 --- [ocp-schedules-24,c47991748a23e353,c0a3f983489a6a0a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:36:21.550 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:21.550 INFO 2458 --- [ocp-schedules-20,9c88958dc07fa3b8,c0f894a8c64a82d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:21.550 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:21.552 INFO 2458 --- [ocp-schedules-24,c47991748a23e353,c0a3f983489a6a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:21.553 INFO 2458 --- [ocp-schedules-24,c47991748a23e353,c0a3f983489a6a0a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:36:21.553 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:21.553 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:21.554 INFO 2458 --- [ocp-schedules-7,87eacb4bdf677ebb,d4c936e13c2ddfb4] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:36:21.555 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:21.556 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:21.556 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:21.557 INFO 2458 --- [ocp-schedules-7,87eacb4bdf677ebb,d4c936e13c2ddfb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:21.558 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:21.559 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:21.565 INFO 2458 --- [ocp-schedules-21,544a81cc29cd9fb5,3efdf4829971df35] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:36:21.604 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.605 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:21.606 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:36:21.607 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.608 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:21.612 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:36:21.612 INFO 2458 --- [ocp-schedules-20,9c88958dc07fa3b8,c0f894a8c64a82d3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:36:21.617 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:21.617 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:21.622 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:21.622 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.623 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:21.623 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:21.627 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:21.637 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:36:21.638 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:36:21.638 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:36:21.640 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:36:21.656 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:36:21.674 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:36:21.676 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:21.676 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:36:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.677 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:36:21.677 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:21.677 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:21.684 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:36:21.692 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:36:21.728 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:21.729 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.730 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:36:21.738 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:21.737576+08:00, lastProcessTime=2025-10-27T11:36:11.714715+08:00 2025-10-27 11:36:21.755 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:21.756 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:21.758 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:21.762 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:21.773 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:36:21.773 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:36:21.774 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:21.777 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:36:21.786 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:36:21.805 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:36:21.805 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:21.806 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.806 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:21.808 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.811 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:36:21.811 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:21.812 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:36:21.812 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:36:21.815 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:21.815 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.816 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:36:21.816 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:36:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:21.839 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:21.844 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:36:21.854 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:21.858 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:21.862 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:21.862 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:21.874 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:21.874 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:36:21.875 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:36:21.877 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:36:21.884 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:21.886 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:36:21.909 INFO 2458 --- [host-check-host-status2,b4f5f0912b86ad8e,71c324b6e216a131] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:36:21.910 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:21.911 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.911 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:36:21.914 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:36:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:21.918 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:21.920 INFO 2458 --- [host-check-host-status5,b4f5f0912b86ad8e,b9eee9862b8a401d] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:36:21.920 INFO 2458 --- [ocp-schedules-18,0734d40a7d8f4f8e,2d2b86143dc45880] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:36:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.921 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:21.923 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:21.923 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:21.923 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:21.925 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:36:21.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:21.928 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:36:21.931 INFO 2458 --- [host-check-host-status4,b4f5f0912b86ad8e,876f3e7c6359f3af] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:36:21.933 INFO 2458 --- [host-check-host-status3,b4f5f0912b86ad8e,f3e0877c8394ece9] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:36:21.940 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:21.943 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:21.956 INFO 2458 --- [host-check-host-status1,b4f5f0912b86ad8e,289f449a6be35fef] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:36:21.968 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:21.968 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:22.006 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:36:22.011 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:22.011 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:22.049 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:36:22.049 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.050 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:36:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:22.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:22.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:36:22.068 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:36:22.094 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:22.095 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.095 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:36:22.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:22.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:22.117 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:36:22.117 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:36:22.131 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:22.131 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:22.208 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:22.208 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.209 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:22.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:36:22.222 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:36:22.231 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:22.234 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:36:22.318 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:21.737576+08:00 2025-10-27 11:36:22.361 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:36:22.361 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.362 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:36:22.364 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.365 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:36:22.382 INFO 2458 --- [host-check-clockdiff4,0c334b299bf01cbf,6cbe789de26e121a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536182 0 0 , error= 2025-10-27 11:36:22.401 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:36:22.448 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:22.449 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:22.449 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:36:22.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:36:22.490 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:36:22.490 INFO 2458 --- [ocp-schedules-7,87eacb4bdf677ebb,d4c936e13c2ddfb4] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:36:22.672 WARN 2458 --- [ocp-async-1,9bdbbf77e82e315d,7809d6b72e178e15] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:22.674 INFO 2458 --- [ocp-async-2,5711a6cae0eb942e,0e739534cda0e6e2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:23.226 INFO 2458 --- [host-check-clockdiff3,0c334b299bf01cbf,547f4029bad56f2a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536183 -1 -1 , error= 2025-10-27 11:36:23.707 INFO 2458 --- [prometheus-minute-schedule-1,7b9b6776028de1aa,3511c047c8c69d93] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:23.721 INFO 2458 --- [prometheus-minute-schedule-1,7b9b6776028de1aa,3511c047c8c69d93] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:24.672 WARN 2458 --- [ocp-async-6,6a9dd0854a06d386,cfcc19909ce52bfd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:24.674 INFO 2458 --- [ocp-async-5,456c560df94f537f,355565bf227f3cc7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:24.677 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:24.678 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672} 2025-10-27 11:36:24.681 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:24.696 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterVi 2025-10-27 11:36:24.697 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:24.698 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:24.698 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:36:24.698 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:24.701 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:24.702 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:24.702 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:36:24.702 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:24.705 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:24.718 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:24.718 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:36:24.718 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:24.860 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449784860000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:36:24.860Z, endTime=2025-10-27T03:34:24.672Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:36:24.863 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449784862000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:36:24.862Z, endTime=2025-10-27T03:34:24.672Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:36:24.877 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:36:24.877 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:36:24.901 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 264 2025-10-27 11:36:24.901 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:36:24.902 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:24.902 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:24.932 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:24.932 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:24.976 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:36:24.976 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:36:24.983 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:36:24.983 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:36:25.000 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:25.000 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:25.006 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:36:25.006 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:25.006 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:25.010 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:36:25.010 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:36:24.86, end=2025-10-27T11:34:24.672}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:36:25.030 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:36:24.862, end=2025-10-27T11:34:24.672}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:36:25.030 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:25.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:25.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:36:25.032 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:25.035 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:25.039 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 2 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:25.051 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:36:25.051 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:25.162 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449785162000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:36:25.162Z, endTime=2025-10-27T03:34:24.672Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:36:25.208 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 260 2025-10-27 11:36:25.208 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:36:25.246 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:25.246 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:25.254 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:36:25.254 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:36:25.279 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:36:25.279 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536064672000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:36:25.284 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:36:25.303 INFO 2458 --- [plan_change_detector6,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:36:25.162, end=2025-10-27T11:34:24.672}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:36:25.304 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:36:25.305 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:32:23.325, end=2025-10-27T11:34:24.672},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:36:25.305 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:36:25.305 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:36:25.308 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:36:25.942 INFO 2458 --- [ocp-schedules-4,38d35dfa95aca122,574d0ef98671c623] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:25.945 INFO 2458 --- [ocp-schedules-4,38d35dfa95aca122,574d0ef98671c623] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:26.672 WARN 2458 --- [ocp-async-8,082741c620c80676,289224f5d39e64ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:26.674 INFO 2458 --- [ocp-async-9,4bf9426d6c3eabfe,c6d5f052d1596e73] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:26.707 INFO 2458 --- [prometheus-minute-schedule-1,00c6bbd66f8e1488,d4a2d6e768c11c32] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:26.718 INFO 2458 --- [prometheus-minute-schedule-1,00c6bbd66f8e1488,d4a2d6e768c11c32] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:27.181 INFO 2458 --- [ocp-schedules-11,98e7a5daebff236c,0d5621324354cd26] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:27.681 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:27.682 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:27.811 INFO 2458 --- [ocp-schedules-6,7928639534b4a4f4,80dc4afb5164de91] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:36:27.823 INFO 2458 --- [ocp-schedules-6,7928639534b4a4f4,80dc4afb5164de91] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:27.823 INFO 2458 --- [ocp-schedules-6,7928639534b4a4f4,80dc4afb5164de91] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:28.427 INFO 2458 --- [ocp-schedules-14,ae4c13a64ee1843b,60452e708b0416c2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:28.475 WARN 2458 --- [scanner-112217709-1,ae4c13a64ee1843b,3674ee995a212513] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:28.543 INFO 2458 --- [ocp-async-3,9a09b94ef5b10abc,c2043ab4b43e1a43] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:28.658 INFO 2458 --- [alarm-detect-6,ae4c13a64ee1843b,9181e84cf94522a7] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536160 ~ 1761536178 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536175, value=74.541)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536175, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536175=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:36:28.672 WARN 2458 --- [ocp-async-7,9b44b5b74412313d,55d8cd199dc5d831] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:28.674 INFO 2458 --- [ocp-async-2,c0b1b28887a4144f,6cd8626d3cc44748] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:29.084 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:36:29.084 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:36:29.090 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:36:29.090 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839265989532524544 2025-10-27 11:36:29.096 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:36:29.096 INFO 2458 --- [ocp-schedules-30,b62748d4df5e72d7,8bb585f67038ac96] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:36:29.526 INFO 2458 --- [ocp-schedules-10,a84f08112122b838,e2b86709c98e969c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:36:29.540 INFO 2458 --- [ocp-schedules-10,a84f08112122b838,e2b86709c98e969c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:36:29.540 INFO 2458 --- [ocp-schedules-10,a84f08112122b838,e2b86709c98e969c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:36:29.707 INFO 2458 --- [prometheus-minute-schedule-1,a09c956729206429,aab215bcf6b2725e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:29.719 INFO 2458 --- [prometheus-minute-schedule-1,a09c956729206429,aab215bcf6b2725e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:30.672 WARN 2458 --- [ocp-async-4,cd119363df27c9d8,ef04bcf503a96c1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:30.674 INFO 2458 --- [ocp-async-5,1c97d45120a1db36,66ea484f2a3f9cf6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:30.682 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:30.689 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:31.714901+08:00, lastProcessTime=2025-10-27T11:36:21.737576+08:00 2025-10-27 11:36:31.972 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:31.714901+08:00 2025-10-27 11:36:32.673 WARN 2458 --- [ocp-async-9,7fba511a84100b4d,10cc9b869f58806b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:32.674 INFO 2458 --- [ocp-async-3,46f0eb85d87c0b23,d447247dc17031cc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:32.707 INFO 2458 --- [prometheus-minute-schedule-1,f3f1bbad8549af32,58ba539166b96359] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:32.717 INFO 2458 --- [prometheus-minute-schedule-1,f3f1bbad8549af32,58ba539166b96359] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:33.690 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:33.691 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:33.998 INFO 2458 --- [ocp-schedules-22,dc9713a5f12ddd32,9f95cc8e36266837] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:36:33.998 INFO 2458 --- [ocp-schedules-22,dc9713a5f12ddd32,9f95cc8e36266837] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:36:34.582 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:36:34.582 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:36:34.605 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:36:34.606 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:36:34.606 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:36:34.620 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:36:34.621 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:36:34.621 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:36:34.645 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:36:34.645 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:36:34.645 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:36:34.657 INFO 2458 --- [ocp-schedules-8,c56b8f8aa4109a2e,80c070b3fafc8270] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:36:34.673 WARN 2458 --- [ocp-async-10,4477d9ff1e5bf5df,7da08ce37197e981] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:34.674 INFO 2458 --- [ocp-async-1,6691f9c3620897a9,4df13455eab15d5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:35.707 INFO 2458 --- [prometheus-minute-schedule-1,f76fd292239799b0,20313df8e64fa036] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:35.714 INFO 2458 --- [prometheus-minute-schedule-1,f76fd292239799b0,20313df8e64fa036] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:35.953 INFO 2458 --- [ocp-schedules-19,3e3d2818d4dc9da4,a875443f1cf3e0a8] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:35.957 INFO 2458 --- [ocp-schedules-19,3e3d2818d4dc9da4,a875443f1cf3e0a8] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:36.673 WARN 2458 --- [ocp-async-2,338db026f3dd71d4,acd06fcdf5e95229] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:36.675 INFO 2458 --- [ocp-async-6,ba2bdaf43f0efa01,8b08f91dd3c5e7fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:36.691 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:36.692 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:37.192 INFO 2458 --- [ocp-schedules-24,415b24b27e9eacc9,188b6fac2f05cbfd] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:37.708 INFO 2458 --- [prometheus-cache-monitor1,bb67c9b81b31dc0d,56640c05d7c8947b] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20317,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:36:37.708 INFO 2458 --- [prometheus-cache-monitor1,bb67c9b81b31dc0d,56640c05d7c8947b] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:36:38.543 INFO 2458 --- [ocp-async-1,c555230076c91429,4fee6f23141e84eb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:38.663 INFO 2458 --- [ocp-schedules-24,43ac052831873c9f,a15a9df91c2bae9e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:38.673 WARN 2458 --- [ocp-async-5,16b0a8d78acd9ac6,3678bb51ff99319e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:38.675 INFO 2458 --- [ocp-async-8,a684c68735ea0609,46f14f8d3e5295ba] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:38.697 WARN 2458 --- [scanner-112217924-1,43ac052831873c9f,bd4068cc0b0e5093] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:38.707 INFO 2458 --- [prometheus-minute-schedule-1,7fb11bec2a2ec289,aeef4b7e21abc674] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:38.715 INFO 2458 --- [prometheus-minute-schedule-1,7fb11bec2a2ec289,aeef4b7e21abc674] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:38.857 INFO 2458 --- [ocp-schedules-19,3e10fc71d7ae1692,8252eba141509a1c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:36:38.864 INFO 2458 --- [ocp-schedules-19,3e10fc71d7ae1692,8252eba141509a1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:38.870 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:36:38.920 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:36:38.935 INFO 2458 --- [ocp-schedules-19,3e10fc71d7ae1692,8252eba141509a1c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:36:39.692 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:39.693 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:40.008 INFO 2458 --- [ocp-schedules-16,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:36:40.008 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:36:40.080 INFO 2458 --- [Collect obproxy1,465681a93c665db9,efaa759a84e0f474] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:36:40.080 INFO 2458 --- [Check obproxy1,66996bfc61e38252,1c4a4e1194427ee6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:36:40.080 INFO 2458 --- [Check obproxy1,66996bfc61e38252,1c4a4e1194427ee6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:36:40.080 INFO 2458 --- [Collect obproxy1,465681a93c665db9,efaa759a84e0f474] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:36:40.141 INFO 2458 --- [Collect obproxy1,465681a93c665db9,efaa759a84e0f474] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:36:40.142 INFO 2458 --- [Check obproxy1,66996bfc61e38252,1c4a4e1194427ee6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:36:40.145 INFO 2458 --- [Collect obproxy1,465681a93c665db9,efaa759a84e0f474] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:36:40.145 INFO 2458 --- [Check obproxy1,66996bfc61e38252,1c4a4e1194427ee6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:36:40.145 INFO 2458 --- [Check obproxy1,66996bfc61e38252,1c4a4e1194427ee6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:36:40.148 INFO 2458 --- [Collect obproxy1,465681a93c665db9,efaa759a84e0f474] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:36:40.248 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:40.248 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:36:40.248 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:36:40.248 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:36:40.248 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:36:40.248 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:36:40.262 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:40.262 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:40.263 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:40.263 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:40.263 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:40.267 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:40.267 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:40.267 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:40.267 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:40.267 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:40.380 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:36:40.383 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:36:40.392 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:36:40.402 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:36:40.402 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:36:40.404 INFO 2458 --- [ocp-schedules-28,6e43ca032dfeb551,ba4263e0d9d3133e] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:36:40.673 WARN 2458 --- [ocp-async-10,fb211f04280b59f8,d8fa52a46a296040] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:40.675 INFO 2458 --- [ocp-async-7,96ef48907adbf1a1,da1fcfcc36572b87] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:40.736 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:36:40.736 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:36:40.832 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:40.832 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:40.881 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:40.881 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:40.881 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:40.882 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:36:40.894 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:40.894 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:40.894 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:40.895 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:36:40.957 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:40.958 INFO 2458 --- [ocp-schedules-18,2491478100c38d20,b553a070ce734d8e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:36:41.296 INFO 2458 --- [ocp-schedules-25,e4acd9913e18a6ba,751bb76e9b49e5c2] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:36:41.707 INFO 2458 --- [prometheus-minute-schedule-1,54e0796c57e21260,19b6be56f506fa1e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:41.713 INFO 2458 --- [prometheus-minute-schedule-1,54e0796c57e21260,19b6be56f506fa1e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:41.721 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:41.719854+08:00, lastProcessTime=2025-10-27T11:36:31.714901+08:00 2025-10-27 11:36:41.918 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:41.719854+08:00 2025-10-27 11:36:42.673 WARN 2458 --- [ocp-async-7,85289e61de5cb186,2500079c99cbac9f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:42.675 INFO 2458 --- [ocp-async-1,ff032148d40da31b,6409b6db0f83a598] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:42.693 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:42.694 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:44.313 INFO 2458 --- [ocp-schedules-21,5f891b9b296010c9,74d39c14e79eb0ec] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:36:44.673 WARN 2458 --- [ocp-async-5,ea74e2b21b3bae33,f8100437096c6e2b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:44.676 INFO 2458 --- [ocp-async-3,37dfddb4d4a4fad9,5ffc696e3d6b42dc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:44.707 INFO 2458 --- [prometheus-minute-schedule-1,4d8314c907207951,6a74aaaf19033671] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:44.714 INFO 2458 --- [prometheus-minute-schedule-1,4d8314c907207951,6a74aaaf19033671] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:45.102 INFO 2458 --- [prometheus-second-schedule-1,802096c700eb2a06,8ed8dfd6f0980757] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:36:45.694 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:45.696 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:45.966 INFO 2458 --- [ocp-schedules-7,68d0bc30f8b405b4,7f83875c86b6d88e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:45.969 INFO 2458 --- [ocp-schedules-7,68d0bc30f8b405b4,7f83875c86b6d88e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:46.297 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,84c154d32f8d2256,1b69406f7334dc65] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=84c154d32f8d2256, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:36:46.298 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,84c154d32f8d2256,1b69406f7334dc65] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=84c154d32f8d2256, duration=1 ms] 2025-10-27 11:36:46.674 WARN 2458 --- [ocp-async-3,f245ce684cd36a5d,6bb63cb1cf9e71a7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:46.676 INFO 2458 --- [ocp-async-2,17feee9010f976ec,74ea376af5008c59] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:47.198 INFO 2458 --- [ocp-schedules-9,56375be39fd3e7a0,a61afec163407d65] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:47.707 INFO 2458 --- [prometheus-minute-schedule-1,a32d372bed064416,41097e1d2be20627] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:47.713 INFO 2458 --- [prometheus-minute-schedule-1,a32d372bed064416,41097e1d2be20627] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:47.826 INFO 2458 --- [ocp-schedules-12,dd811a1f0896ae65,4ba14ddd3037ac24] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:36:47.848 INFO 2458 --- [ocp-schedules-12,dd811a1f0896ae65,4ba14ddd3037ac24] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:47.848 INFO 2458 --- [ocp-schedules-12,dd811a1f0896ae65,4ba14ddd3037ac24] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:36:48.543 INFO 2458 --- [ocp-async-8,54d1ea9bd914b47d,822e3f7fc2cd556c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:48.674 WARN 2458 --- [ocp-async-6,696e4aadf1a1b99d,5f3835cd08781d7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:48.676 INFO 2458 --- [ocp-async-7,e18dd7baf9c88ced,84d2da81bd85eb54] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:48.696 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:48.696 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:48.942 INFO 2458 --- [ocp-schedules-3,b350f32723b2cecc,62ed476d4361befc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:48.984 WARN 2458 --- [scanner-112218113-1,b350f32723b2cecc,ff57bf43a39a3f66] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:49.112 INFO 2458 --- [alarm-detect-8,b350f32723b2cecc,904425104c9d616a] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536139 ~ 1761536199 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536160, value=204494.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204494.56, values={1761536160=204494.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:36:49.113 INFO 2458 --- [alarm-detect-8,b350f32723b2cecc,904425104c9d616a] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:36:49.143 INFO 2458 --- [alarm-detect-1,b350f32723b2cecc,7f8ff1dfca0cb58c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536139 ~ 1761536199 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536160, value=93.242)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.242, values={1761536160=93.242}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:36:49.144 INFO 2458 --- [alarm-detect-1,b350f32723b2cecc,7f8ff1dfca0cb58c] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:36:49.166 INFO 2458 --- [alarm-detect-7,b350f32723b2cecc,d3c86d3e6edea569] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536139 ~ 1761536199 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536160, value=204495.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204495.912, values={1761536160=204495.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:36:49.167 INFO 2458 --- [alarm-detect-7,b350f32723b2cecc,d3c86d3e6edea569] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:36:49.187 INFO 2458 --- [alarm-detect-15,b350f32723b2cecc,b1c81727338c6c76] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536175 ~ 1761536199 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536190, value=75.688)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536190, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536190=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:36:50.113 INFO 2458 --- [ocp-schedules-23,8d5a14aaa8ecf54b,2467a125cf6ac722] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:36:50.113+08:00, clusters=[1, 2] 2025-10-27 11:36:50.127 INFO 2458 --- [ocp-schedules-23,8d5a14aaa8ecf54b,2467a125cf6ac722] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:36:50.130 INFO 2458 --- [ocp-schedules-23,8d5a14aaa8ecf54b,2467a125cf6ac722] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:36:50.674 WARN 2458 --- [ocp-async-1,317f8e5e1b37b5b9,51d00d212e13f412] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:50.676 INFO 2458 --- [ocp-async-9,6e091db75cf98068,be0d095cd3c89a9d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:50.707 INFO 2458 --- [prometheus-minute-schedule-1,1c782d253dc542b2,0b8852d8fb3f7936] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:50.714 INFO 2458 --- [prometheus-minute-schedule-1,1c782d253dc542b2,0b8852d8fb3f7936] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:51.433 INFO 2458 --- [ocp-schedules-18,bd8e803012c68f75,e76840eba23bf612] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:51.433 INFO 2458 --- [ocp-schedules-17,c97fb58e635b30f1,946861d85557c808] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:51.433 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:36:51.434 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:36:51.435 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:36:51.435 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:36:51.437 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:36:51.442 INFO 2458 --- [ocp-schedules-11,0c707dbf44ba3fdb,b44a1e6a70498c9a] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:36:51.449 INFO 2458 --- [ocp-schedules-22,11edf1b0b879fc22,6ab927291de21ad7] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:36:51.449 INFO 2458 --- [host-check-clockdiff1,11edf1b0b879fc22,630c5247e1aef43c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-clockdiff2,11edf1b0b879fc22,29e223ecd205a983] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-clockdiff3,11edf1b0b879fc22,4e0dfbae345d3679] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:51.450 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:51.451 INFO 2458 --- [host-check-clockdiff4,11edf1b0b879fc22,50405347fc17aa9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:36:51.451 INFO 2458 --- [host-check-clockdiff5,11edf1b0b879fc22,73b37a4c8c003439] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:36:51.451 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:51.473 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:51.476 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:51.477 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:51.478 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:51.479 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:51.485 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:51.485 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:51.485 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:51.485 INFO 2458 --- [host-check-clockdiff4,11edf1b0b879fc22,50405347fc17aa9f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536211 0 0 , error= 2025-10-27 11:36:51.486 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:51.488 INFO 2458 --- [host-check-clockdiff1,11edf1b0b879fc22,630c5247e1aef43c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536211 0 0 , error= 2025-10-27 11:36:51.488 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:51.491 INFO 2458 --- [host-check-clockdiff2,11edf1b0b879fc22,29e223ecd205a983] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536211 -1 -1 , error= 2025-10-27 11:36:51.492 INFO 2458 --- [host-check-clockdiff5,11edf1b0b879fc22,73b37a4c8c003439] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536211 0 0 , error= 2025-10-27 11:36:51.493 INFO 2458 --- [host-check-clockdiff3,11edf1b0b879fc22,4e0dfbae345d3679] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536211 0 0 , error= 2025-10-27 11:36:51.545 INFO 2458 --- [ocp-schedules-21,5c6200cc1becb5a4,17e75b3b89de87f3] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:36:51.547 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:36:51.549 INFO 2458 --- [ocp-schedules-24,3da247b8c7c71d42,57d1549ee8fc8e45] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:36:51.551 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:36:51.551 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:36:51.551 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:36:51.551 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:36:51.551 INFO 2458 --- [host-check-agent-status5,c97fb58e635b30f1,a0aaaa229d2db6dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:36:51.552 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:36:51.552687085+08:00 2025-10-27 11:36:51.553 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:36:51.553+08:00 2025-10-27 11:36:51.554 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:36:51.554 INFO 2458 --- [ocp-schedules-13,a07f33597e06d284,db82b04f9a492d31] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:36:51.554 INFO 2458 --- [ocp-schedules-21,5c6200cc1becb5a4,17e75b3b89de87f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:51.554 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:36:51.555 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:36:51.555 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:36:51.555 INFO 2458 --- [host-check-agent-status3,c97fb58e635b30f1,f9fecafc817284fd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:36:51.555 INFO 2458 --- [ocp-schedules-24,3da247b8c7c71d42,57d1549ee8fc8e45] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:51.556 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:36:51.555583443+08:00 2025-10-27 11:36:51.556 INFO 2458 --- [ocp-schedules-24,3da247b8c7c71d42,57d1549ee8fc8e45] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:36:51.557 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:36:51.556+08:00 2025-10-27 11:36:51.558 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:36:51.558 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:36:51.558 INFO 2458 --- [host-check-agent-status1,c97fb58e635b30f1,4a75780baeeeafd0] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:36:51.558 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:36:51.559 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:36:51.559 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:36:51.561 INFO 2458 --- [ocp-schedules-13,a07f33597e06d284,db82b04f9a492d31] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:36:51.562 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:36:51.560841114+08:00 2025-10-27 11:36:51.562 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:36:51.562 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:36:51.561964718+08:00 2025-10-27 11:36:51.562 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:36:51.562+08:00 2025-10-27 11:36:51.563 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:36:51.562+08:00 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status4,c97fb58e635b30f1,759f5c66c0789fc9] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-agent-status2,c97fb58e635b30f1,fab17552ad96faab] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:36:51.564 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:36:51.553+08:00 and status AVAILABLE 2025-10-27 11:36:51.565 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:36:51.556+08:00 and status AVAILABLE 2025-10-27 11:36:51.568 INFO 2458 --- [ocp-schedules-19,f74275bbe5edc6aa,4eba4961a3f2126c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:36:51.569 INFO 2458 --- [ocp-schedules-7,7c1607d2eefa5087,5152f6f321049b2e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:36:51.569 INFO 2458 --- [ocp-schedules-9,1c50ecb9aa7e7d2f,8df976ae407d2d5c] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:36:51.570 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:36:51.571 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:36:51.562+08:00 and status AVAILABLE 2025-10-27 11:36:51.571 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:36:51.562+08:00 and status AVAILABLE 2025-10-27 11:36:51.573 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:36:51.573 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:36:51.574 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:36:51.574039957+08:00 2025-10-27 11:36:51.574 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:36:51.574+08:00 2025-10-27 11:36:51.576 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:36:51.580 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:36:51.580 INFO 2458 --- [ocp-schedules-19,f74275bbe5edc6aa,4eba4961a3f2126c] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:36:51.582 INFO 2458 --- [ocp-schedules-7,7c1607d2eefa5087,5152f6f321049b2e] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:36:51.589 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:36:51.574+08:00 and status AVAILABLE 2025-10-27 11:36:51.594 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:36:51.604 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:36:51.604 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:36:51.606 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.607 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:36:51.607 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:51.607 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:36:51.607 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:36:51.608 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:36:51.609 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:36:51.616 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:51.616 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:51.618 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:51.619 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.622 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.623 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:51.627 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:36:51.627 INFO 2458 --- [ocp-schedules-21,5c6200cc1becb5a4,17e75b3b89de87f3] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:36:51.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.639 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:51.666 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:51.670 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:51.674 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:51.678 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:51.682 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:36:51.688 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:36:51.694 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:36:51.695 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:36:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.696 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:51.697 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:51.698 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:36:51.704 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:36:51.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:36:51.715711+08:00, lastProcessTime=2025-10-27T11:36:41.719854+08:00 2025-10-27 11:36:51.723 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:36:51.730 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:51.730 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.731 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:36:51.731 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:36:51.735 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:36:51.735 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:36:51.746 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:36:51.766 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:51.769 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:51.770 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:51.770 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:51.774 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:36:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.780 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.781 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.781 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:36:51.781 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:36:51.788 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:36:51.797 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:36:51.798 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:36:51.800 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:36:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:51.821 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.822 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:51.823 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:36:51.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.826 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:36:51.826 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:36:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:51.830 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:36:51.835 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:36:51.838 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:36:51.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:51.860 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:51.872 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:51.875 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:51.876 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:51.876 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.881 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:36:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:51.883 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:51.884 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.884 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:36:51.886 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:36:51.888 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:51.888 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:36:51.890 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:36:51.895 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:36:51.903 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:51.905 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:36:51.910 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:36:51.925 INFO 2458 --- [ocp-schedules-3,6964a4544b1f5ee2,5ea3807916ee3042] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:36:51.932 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:51.933 INFO 2458 --- [host-check-host-status5,bd8e803012c68f75,77a5a54b4fc2f08e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:36:51.933 INFO 2458 --- [host-check-host-status2,bd8e803012c68f75,990263d28810d8d3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:36:51.933 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:51.934 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:51.934 INFO 2458 --- [host-check-host-status1,bd8e803012c68f75,0da0cfbc902fd9ff] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:36:51.934 INFO 2458 --- [host-check-host-status4,bd8e803012c68f75,4432f69ab965baf5] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:36:51.935 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:36:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:51.937 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:51.940 INFO 2458 --- [host-check-host-status3,bd8e803012c68f75,c38e4f9abebae8b3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:36:51.949 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:36:51.959 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:36:51.962 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:36:51.983 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:51.983 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:51.994 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:36:51.715711+08:00 2025-10-27 11:36:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:52.018 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:52.019 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:36:52.059 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:52.059 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:36:52.063 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:36:52.065 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.066 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:36:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.068 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:36:52.086 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:36:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:52.092 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:36:52.107 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:52.108 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.109 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:36:52.128 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:36:52.133 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:36:52.133 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:36:52.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:52.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.165 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.166 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.166 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:36:52.170 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:36:52.181 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:36:52.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:36:52.193 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:36:52.238 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:36:52.286 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:36:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.286 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.287 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:36:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.289 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.289 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:36:52.321 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:36:52.362 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:36:52.363 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:36:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:36:52.403 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:36:52.403 INFO 2458 --- [ocp-schedules-13,a07f33597e06d284,db82b04f9a492d31] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:36:52.674 WARN 2458 --- [ocp-async-8,5b7229cf5362465f,dbdb0d15729bd7ba] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:52.676 INFO 2458 --- [ocp-async-6,27326246dbafbf9f,e426a1e0ee8acb1e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:36:53.707 INFO 2458 --- [prometheus-minute-schedule-1,21825c38bcebc455,de86a728b205448e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:53.714 INFO 2458 --- [prometheus-minute-schedule-1,21825c38bcebc455,de86a728b205448e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:54.674 WARN 2458 --- [ocp-async-4,374b79dc9a9e3d2b,76aa625ccf624e9e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:54.676 INFO 2458 --- [ocp-async-1,08ffa8cf0dedb0cf,73a9801a31c13bac] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:54.697 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:54.698 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:55.803 INFO 2458 --- [ocp-schedules-28,88bc0466201d73e3,4bedc65de11eba49] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:36:55.980 INFO 2458 --- [ocp-schedules-24,9949726fc356959b,eb93d0379c57f93c] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:36:55.983 INFO 2458 --- [ocp-schedules-24,9949726fc356959b,eb93d0379c57f93c] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:36:56.674 WARN 2458 --- [ocp-async-7,1991de71f8fedfaf,1580039b06e98cff] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:56.676 INFO 2458 --- [ocp-async-3,454d74c1975ecb64,b12d6fdd03f77b2f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:56.707 INFO 2458 --- [prometheus-minute-schedule-1,42436eaca4a09156,2653d8075496812c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:56.714 INFO 2458 --- [prometheus-minute-schedule-1,42436eaca4a09156,2653d8075496812c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:36:57.204 INFO 2458 --- [ocp-schedules-14,cf962f8e91fb27bb,fc53656e1aa60e20] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:36:57.698 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:36:57.699 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:36:58.543 INFO 2458 --- [ocp-async-3,47fd01fb6fd08502,54355cbd42900426] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:36:58.675 WARN 2458 --- [ocp-async-8,bee0fe6ba2b168d3,4f4bc7c20124cd82] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:36:58.676 INFO 2458 --- [ocp-async-6,a1957f8503332310,661994ca25eb9d25] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:36:59.197 INFO 2458 --- [ocp-schedules-16,45160b6f09a287a5,85acc79de7133924] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:36:59.252 WARN 2458 --- [scanner-112218385-1,45160b6f09a287a5,400f33a44f0c7700] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:36:59.382 INFO 2458 --- [alarm-detect-15,45160b6f09a287a5,169cdae09ab01ce8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536190 ~ 1761536209 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536205, value=77.314)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536205, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536205=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:36:59.592 INFO 2458 --- [ocp-schedules-9,1ee2caab0c37b875,0be4674d49b129cb] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:36:59.625 INFO 2458 --- [ocp-schedules-5,d05b0bf214ec6d5f,07e1d59678f67c24] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:36:59.630 INFO 2458 --- [ocp-schedules-5,d05b0bf214ec6d5f,07e1d59678f67c24] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:36:59.707 INFO 2458 --- [prometheus-minute-schedule-1,0aa72d882ea384cf,4d2923b44ed351c6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536220, interval=60 2025-10-27 11:36:59.714 INFO 2458 --- [prometheus-minute-schedule-1,0aa72d882ea384cf,4d2923b44ed351c6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:00.675 WARN 2458 --- [ocp-async-5,47de323c87fc3307,6abec1f39efaf6d3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:00.677 INFO 2458 --- [ocp-async-1,0e0851074783205f,57f9594399fde459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:00.700 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:00.701 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:01.714490+08:00, lastProcessTime=2025-10-27T11:36:51.715711+08:00 2025-10-27 11:37:01.923 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:01.714490+08:00 2025-10-27 11:37:02.675 WARN 2458 --- [ocp-async-9,c3efbd1d17c7544f,7d98efaa35be8737] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:02.677 INFO 2458 --- [ocp-async-3,367da4c4c563e2a7,5124179f9d89e662] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:02.707 INFO 2458 --- [prometheus-minute-schedule-1,bf8c5ac3d506018a,bf172cf8795db236] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:02.714 INFO 2458 --- [prometheus-minute-schedule-1,bf8c5ac3d506018a,bf172cf8795db236] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:03.143 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,ee9765f255d8d494,25a98e9db00ad33e] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=ee9765f255d8d494, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:37:03.143 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,ee9765f255d8d494,25a98e9db00ad33e] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=ee9765f255d8d494, duration=1 ms] 2025-10-27 11:37:03.701 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:03.702 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:04.675 WARN 2458 --- [ocp-async-2,ce169c2e669504c7,46f92e0cb6d40209] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:04.677 INFO 2458 --- [ocp-async-6,c86d66c94e8284b8,b50fa1ecd8334271] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:04.983 INFO 2458 --- [ocp-schedules-14,05aaf1a4f7273a6d,2fd82d5f27213286] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:37:04.990 INFO 2458 --- [ocp-schedules-14,05aaf1a4f7273a6d,2fd82d5f27213286] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:05.039 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:05.040 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:37:05.050 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:05.050 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:05.050 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:05.050 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:05.051 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:05.051 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:37:05.065 INFO 2458 --- [ocp-schedules-14,05aaf1a4f7273a6d,2fd82d5f27213286] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:37:05.707 INFO 2458 --- [prometheus-minute-schedule-1,2ffd17dbd9af19f6,83d0f1fae4bd3d6b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:05.714 INFO 2458 --- [prometheus-minute-schedule-1,2ffd17dbd9af19f6,83d0f1fae4bd3d6b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:05.991 INFO 2458 --- [ocp-schedules-26,6e7aa86bc9551b4d,92cf777d0ccd1041] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:05.994 INFO 2458 --- [ocp-schedules-26,6e7aa86bc9551b4d,92cf777d0ccd1041] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:06.675 WARN 2458 --- [ocp-async-4,67edf93418eeaf18,986d8ad6864057cc] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:06.677 INFO 2458 --- [ocp-async-1,db186f67a3403456,a16e8bc0711bb655] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:06.702 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:06.703 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:07.210 INFO 2458 --- [ocp-schedules-15,7ada6afc5751c2d2,8481d5ee11511b96] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:07.850 INFO 2458 --- [ocp-schedules-12,a9a9e95287178c24,868f417a2b904df8] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:37:07.858 INFO 2458 --- [ocp-schedules-12,a9a9e95287178c24,868f417a2b904df8] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:07.858 INFO 2458 --- [ocp-schedules-12,a9a9e95287178c24,868f417a2b904df8] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:08.543 INFO 2458 --- [ocp-async-1,f040c5d858854c50,155c4188adb5645e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:08.675 WARN 2458 --- [ocp-async-9,1b279c108a03764a,79d48f9e9d717c5d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:08.677 INFO 2458 --- [ocp-async-3,4e7aad1c153edcf6,e6f8da57f8385f20] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:08.707 INFO 2458 --- [prometheus-minute-schedule-1,a7a8dd1ccf070a77,3d1906e1f81c4d50] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:08.713 INFO 2458 --- [prometheus-minute-schedule-1,a7a8dd1ccf070a77,3d1906e1f81c4d50] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:09.388 INFO 2458 --- [ocp-schedules-20,5763a07c67c1b176,c884c339a940ef70] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:37:09.431 WARN 2458 --- [scanner-112218585-1,5763a07c67c1b176,2352fbfde4259723] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:37:09.703 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:09.704 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:10.008 INFO 2458 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:37:10.008 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:37:10.042 INFO 2458 --- [Check obproxy1,474747b225d3dd7b,7028962f92cff8bf] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:37:10.042 INFO 2458 --- [Check obproxy1,474747b225d3dd7b,7028962f92cff8bf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:37:10.043 INFO 2458 --- [Collect obproxy1,9823989d7976013f,25fd251ef0f352df] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:37:10.043 INFO 2458 --- [Collect obproxy1,9823989d7976013f,25fd251ef0f352df] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:37:10.072 INFO 2458 --- [Collect obproxy1,9823989d7976013f,25fd251ef0f352df] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:37:10.072 INFO 2458 --- [Check obproxy1,474747b225d3dd7b,7028962f92cff8bf] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:37:10.075 INFO 2458 --- [Collect obproxy1,9823989d7976013f,25fd251ef0f352df] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:37:10.075 INFO 2458 --- [Check obproxy1,474747b225d3dd7b,7028962f92cff8bf] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:37:10.075 INFO 2458 --- [Check obproxy1,474747b225d3dd7b,7028962f92cff8bf] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:37:10.077 INFO 2458 --- [Collect obproxy1,9823989d7976013f,25fd251ef0f352df] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:37:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:37:10.675 WARN 2458 --- [ocp-async-2,9b83c3f8761fdef2,f81fb237dfa8f0f6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:10.677 INFO 2458 --- [ocp-async-6,d67f6573841a010e,0aa820c0cfe6c935] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:10.825 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=1000, affectRows=1000 2025-10-27 11:37:10.829 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=994, validCount=992 2025-10-27 11:37:10.936 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=1000, affectRows=1000 2025-10-27 11:37:10.941 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=999, validCount=995 2025-10-27 11:37:10.960 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505683) Duplicate entry '835370602979598336-1761536100' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505683) Duplicate entry '835370602979598336-1761536100' for key 'PRIMARY' 2025-10-27 11:37:11.707 INFO 2458 --- [prometheus-minute-schedule-1,273e60b657042a06,a4bb977a78e35fb0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:11.714 INFO 2458 --- [prometheus-minute-schedule-1,273e60b657042a06,a4bb977a78e35fb0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:11.713278+08:00, lastProcessTime=2025-10-27T11:37:01.714490+08:00 2025-10-27 11:37:11.922 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:11.713278+08:00 2025-10-27 11:37:12.676 WARN 2458 --- [ocp-async-4,afa2c2049c153893,6d130a365c7a1a7b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:12.677 INFO 2458 --- [ocp-async-1,91dc3ccfb173c1a0,8b1ec4922b38b255] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:12.704 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:12.705 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:14.676 WARN 2458 --- [ocp-async-7,c8e0eb0e00165cd3,ac5f0557ae634b1b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:14.678 INFO 2458 --- [ocp-async-10,9afdffe6f6c4b906,3afb870ce4f4c8ee] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:14.707 INFO 2458 --- [prometheus-minute-schedule-1,85c5dee74d956e3c,82dd3afa76f62054] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:14.714 INFO 2458 --- [prometheus-minute-schedule-1,85c5dee74d956e3c,82dd3afa76f62054] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:15.660 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:37:15.660 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:37:15.672 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.672 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:37:15.686 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.686 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.688 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.688 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:37:15.696 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.696 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.697 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.698 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:37:15.703 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:37:15.703 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:37:15.705 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:15.705 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.706 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:37:15.706 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:15.708 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.708 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.709 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.710 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:37:15.731 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.732 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.732 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.733 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:37:15.749 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.749 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.750 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.750 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:37:15.764 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.765 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.766 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.766 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:37:15.780 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.781 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.781 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.782 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:37:15.797 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.797 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.798 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.799 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:37:15.814 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.815 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.816 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.816 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:37:15.831 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.831 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.832 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.832 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:37:15.840 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.840 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.841 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.841 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:37:15.850 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.850 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.851 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.852 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:37:15.858 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.858 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.859 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.860 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:37:15.866 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.866 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.867 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.867 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:37:15.874 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.875 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.876 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.876 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:37:15.884 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.884 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.885 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.885 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:37:15.892 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.893 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.894 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.894 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:37:15.901 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.901 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.902 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.903 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:37:15.909 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.909 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.910 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.911 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:37:15.920 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.921 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.922 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.922 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:37:15.933 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.933 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.934 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.934 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:37:15.945 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.946 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.947 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.947 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:37:15.960 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.960 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.961 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.962 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:37:15.968 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.969 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.970 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.970 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:37:15.979 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.979 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.980 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.981 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:37:15.987 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.987 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.988 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.989 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:37:15.994 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.995 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:15.996 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:15.996 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:37:16.003 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.003 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.004 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.004 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:37:16.005 INFO 2458 --- [ocp-schedules-30,1d36a5522af0890d,6d03f4014abaffd2] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:16.011 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.011 INFO 2458 --- [ocp-schedules-30,1d36a5522af0890d,6d03f4014abaffd2] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:16.011 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.012 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.013 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:37:16.019 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.020 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.021 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.021 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:37:16.028 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.028 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.029 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.030 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:37:16.036 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.037 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.037 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.038 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:37:16.043 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.044 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.044 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.045 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:37:16.051 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.051 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.052 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.052 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:37:16.058 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.059 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.059 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.060 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:37:16.066 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.067 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.068 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.069 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:37:16.076 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.076 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.077 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.078 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:37:16.083 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.084 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.085 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.085 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:37:16.090 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.091 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.092 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.092 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:37:16.102 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.102 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.103 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.103 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:37:16.109 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.109 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.110 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.110 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:37:16.116 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.117 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.117 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.118 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:37:16.124 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.125 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.126 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.126 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:37:16.132 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.133 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.133 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.134 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:37:16.145 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.146 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.147 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.147 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:37:16.156 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.157 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.157 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.158 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:37:16.172 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.173 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.174 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.174 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:37:16.182 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.183 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.184 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.184 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:37:16.191 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.191 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.192 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.192 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:37:16.198 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.199 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.199 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.200 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:37:16.207 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.207 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.208 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.208 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:37:16.213 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.214 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.215 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.215 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:37:16.220 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.221 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.222 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.222 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:37:16.228 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.228 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.229 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.230 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:37:16.236 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.236 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.237 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.238 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:37:16.244 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.244 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.245 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.246 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:37:16.251 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.251 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.252 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.253 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:37:16.256 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.257 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.258 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.258 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:37:16.263 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.264 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:37:16.265 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.265 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:37:16.269 INFO 2458 --- [ocp-schedules-21,4b286f71998e968e,c217743c2e45c4c9] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:37:16.502 INFO 2458 --- [ocp-schedules-26,5a2cf50f2d81d867,edd517cac4d0e9c0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:37:16.502+08:00, clusters=[1, 2] 2025-10-27 11:37:16.504 INFO 2458 --- [ocp-schedules-26,5a2cf50f2d81d867,edd517cac4d0e9c0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:37:16.505 INFO 2458 --- [ocp-schedules-26,5a2cf50f2d81d867,edd517cac4d0e9c0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:37:16.676 WARN 2458 --- [ocp-async-2,f9499d649f506dc7,7a44ca076e04a605] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:16.678 INFO 2458 --- [ocp-async-8,2da1c1c059553aa5,615b7235c918a7a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:16.726 INFO 2458 --- [ocp-schedules-22,b09188edd500adb3,85c94f3a8ae8b680] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:37:16.733 INFO 2458 --- [ocp-schedules-22,b09188edd500adb3,85c94f3a8ae8b680] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:16.825 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:37:16.825 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.828 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:37:16.837 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.838 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.863 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.864 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.869 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.869 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.870 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:37:16.870 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.870 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.870 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.874 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:37:16.888 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.889 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.889 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.889 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.889 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.890 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.891 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.891 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:16.916 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.916 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:16.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.918 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.919 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.919 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.943 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.944 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:16.949 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.949 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.949 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.949 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:37:16.951 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.951 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.951 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.952 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:16.976 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.977 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:16.977 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.978 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:16.979 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.980 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:16.980 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.981 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:16.982 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.982 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:16.982 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:16.983 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:16.983 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.983 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:16.983 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:16.984 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:16.985 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:37:16.987 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:37:17.010 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.010 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.011 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.011 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:37:17.013 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.013 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.014 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.014 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:37:17.043 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.043 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.043 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.043 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:37:17.061 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:17.062 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.063 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:37:17.071 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.071 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.072 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:37:17.074 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.074 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.074 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.075 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:37:17.075 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:37:17.075 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:37:17.102 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.102 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.102 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.103 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:37:17.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.130 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.130 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.132 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:37:17.195 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.195 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.196 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:17.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.197 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.197 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.198 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:17.198 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.198 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.199 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.199 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:17.200 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.200 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.200 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.201 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:17.201 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.201 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.202 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.202 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:17.203 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.203 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.203 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.204 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:17.204 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.204 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.205 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.205 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:17.206 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.206 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.206 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.207 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:17.207 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.207 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.207 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.208 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:17.209 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.209 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.209 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.210 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:37:17.210 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.210 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.210 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.211 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:37:17.211 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.211 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.212 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.212 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:37:17.213 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:37:17.215 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:37:17.216 INFO 2458 --- [ocp-schedules-3,47f9911ade859af3,f4d8efec036f7682] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:17.257 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=1000, affectRows=1000 2025-10-27 11:37:17.261 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=996, validCount=992 2025-10-27 11:37:17.286 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=81, affectRows=81 2025-10-27 11:37:17.287 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=80, validCount=80 2025-10-27 11:37:17.298 INFO 2458 --- [metric-archive-async-1,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536160, nodeCount=761, affectRows=761 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.315 INFO 2458 --- [metric-archive-async-8,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536160, nodeCount=60, affectRows=60 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:17.315 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:17.316 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:37:17.337 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=1000, affectRows=1000 2025-10-27 11:37:17.341 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=996, validCount=992 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:37:17.342 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:37:17.346 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:37:17.346 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:37:17.346 INFO 2458 --- [ocp-schedules-22,b09188edd500adb3,85c94f3a8ae8b680] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:37:17.707 INFO 2458 --- [prometheus-minute-schedule-1,c50f7e0c6411bf28,9b16aadddf64200a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:17.714 INFO 2458 --- [prometheus-minute-schedule-1,c50f7e0c6411bf28,9b16aadddf64200a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:18.544 INFO 2458 --- [ocp-async-3,c92d065df568aace,5fda23f979885446] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:18.586 INFO 2458 --- [ocp-schedules-6,756cc7d932469425,a8248117cdc14f68] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:37:18.586 INFO 2458 --- [ocp-schedules-6,756cc7d932469425,a8248117cdc14f68] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:37:18.637 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:37:18.637 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:37:18.638 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:37:18.638 INFO 2458 --- [Collect all tenants major compactions5,756cc7d932469425,3a6339346aba2c19] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:37:18.639 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:37:18.639 INFO 2458 --- [Collect all tenants major compactions6,756cc7d932469425,3b3e9fa908e88c46] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:37:18.671 INFO 2458 --- [Collect all tenants major compactions5,756cc7d932469425,3a6339346aba2c19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.673 INFO 2458 --- [Collect all tenants major compactions5,756cc7d932469425,3a6339346aba2c19] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:37:18.675 INFO 2458 --- [Collect all tenants major compactions6,756cc7d932469425,3b3e9fa908e88c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.676 WARN 2458 --- [ocp-async-5,88426bedfa829d59,1e17b1f4dd339b57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:18.676 INFO 2458 --- [Collect all tenants major compactions6,756cc7d932469425,3b3e9fa908e88c46] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:37:18.678 INFO 2458 --- [ocp-async-1,56adfd9fab52e1a1,cbeb01d77b3d1483] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:18.680 INFO 2458 --- [Collect all tenants major compactions6,756cc7d932469425,3b3e9fa908e88c46] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:37:18.680 INFO 2458 --- [Collect all tenants major compactions5,756cc7d932469425,3a6339346aba2c19] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:37:18.691 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.691 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.691 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.691 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.692 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.693 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:37:18.694 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:37:18.694 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:37:18.695 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.695 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.695 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:18.695 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:18.695 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:18.696 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:37:18.697 INFO 2458 --- [Collect all tenants major compactions3,756cc7d932469425,63e0a21d24383cff] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:37:18.697 INFO 2458 --- [Collect all tenants major compactions1,756cc7d932469425,a49235bfdae1f930] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:37:18.697 INFO 2458 --- [Collect all tenants major compactions2,756cc7d932469425,32f1af68904928c7] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:37:18.699 INFO 2458 --- [Collect all tenants major compactions4,756cc7d932469425,0a7a2fb974c66249] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:37:18.700 INFO 2458 --- [ocp-schedules-6,756cc7d932469425,a8248117cdc14f68] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:37:18.706 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:18.707 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:19.569 INFO 2458 --- [ocp-schedules-24,54204a37616aa0dd,0d05b8612e3f1bc0] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:37:19.600 WARN 2458 --- [scanner-112220390-1,54204a37616aa0dd,256c91aaecf20a5b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:37:19.654 INFO 2458 --- [alarm-detect-12,54204a37616aa0dd,2c552b01f76892d8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536205 ~ 1761536229 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536220, value=1.526)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.526, values={1761536220=1.526}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:37:19.655 INFO 2458 --- [alarm-detect-12,54204a37616aa0dd,2c552b01f76892d8] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:37:19.693 INFO 2458 --- [alarm-detect-18,54204a37616aa0dd,cf57b9a4865fcb4c] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536205 ~ 1761536229 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536220, value=95.184)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536220, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536220=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:37:20.299 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536160, nodeCount=1000, affectRows=1000 2025-10-27 11:37:20.303 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536160, nodeCount=1000, validCount=987 2025-10-27 11:37:20.676 WARN 2458 --- [ocp-async-9,5e627f5296d7e511,00b0ec4ffb1f460b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:20.678 INFO 2458 --- [ocp-async-10,4df5728cfaa4e4f5,d1741e2c2cbfd9d4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:20.707 INFO 2458 --- [prometheus-minute-schedule-1,60ae11e622a2885f,8ea9a93ea232b579] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:20.714 INFO 2458 --- [prometheus-minute-schedule-1,60ae11e622a2885f,8ea9a93ea232b579] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:20.870 INFO 2458 --- [ocp-schedules-19,2807ff6d303ec45f,8692adb9cbc0d93e] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:37:21.409 INFO 2458 --- [ocp-schedules-9,7036af736834c4af,e9230b405a1b1ec1] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:21.409 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:37:21.409 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:37:21.409 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:37:21.409 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:37:21.410 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:37:21.411 INFO 2458 --- [ocp-schedules-23,5344f8921b2aa0ed,3ab8a6d010c5b256] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:21.411 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:37:21.411 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:37:21.411 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:37:21.411 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:37:21.412 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:37:21.422 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:21.423 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:21.424 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:21.424 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:21.425 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:21.425 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:21.425 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:21.425 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:21.427 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:21.428 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:21.428 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:21.428 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:21.428 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:21.429 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:21.429 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:21.429 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:21.429 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:21.429 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:21.431 INFO 2458 --- [ocp-schedules-8,a318ba62e698f44a,e487fe313ccb788e] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:21.431 INFO 2458 --- [host-check-clockdiff1,a318ba62e698f44a,931858802478c7f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:37:21.431 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:21.432 INFO 2458 --- [host-check-clockdiff2,a318ba62e698f44a,20dcebda97ed730d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:37:21.432 INFO 2458 --- [host-check-clockdiff3,a318ba62e698f44a,60996c43e1ab7616] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:37:21.433 INFO 2458 --- [host-check-clockdiff5,a318ba62e698f44a,1b7a6cebcc045ffc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:37:21.433 INFO 2458 --- [host-check-clockdiff4,a318ba62e698f44a,09dc2099c3e3b3bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:37:21.433 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:21.447 INFO 2458 --- [host-check-clockdiff1,a318ba62e698f44a,931858802478c7f0] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536241 0 0 , error= 2025-10-27 11:37:21.453 INFO 2458 --- [ocp-schedules-18,31b5958daa1f48df,3c269ce15f3e7c13] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:37:21.469 INFO 2458 --- [host-check-clockdiff2,a318ba62e698f44a,20dcebda97ed730d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536241 -1 -1 , error= 2025-10-27 11:37:21.506 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:37:21.506 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:37:21.508 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:37:21.507672944+08:00 2025-10-27 11:37:21.508 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:37:21.506984310+08:00 2025-10-27 11:37:21.508 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:37:21.509 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:37:21.508+08:00 2025-10-27 11:37:21.510 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:37:21.510 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:37:21.508+08:00 2025-10-27 11:37:21.511 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:37:21.511483339+08:00 2025-10-27 11:37:21.511 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:37:21.512 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:37:21.511+08:00 2025-10-27 11:37:21.513 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:37:21.513 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:37:21.513 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:37:21.513 INFO 2458 --- [host-check-agent-status1,5344f8921b2aa0ed,3734ad742eb65ebb] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:37:21.514 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:37:21.513844346+08:00 2025-10-27 11:37:21.515 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:37:21.514+08:00 2025-10-27 11:37:21.515 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:37:21.515 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:37:21.515 INFO 2458 --- [host-check-agent-status5,5344f8921b2aa0ed,24914d6402b56523] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:37:21.515 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:37:21.518 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:37:21.508+08:00 and status AVAILABLE 2025-10-27 11:37:21.519 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:37:21.511+08:00 and status AVAILABLE 2025-10-27 11:37:21.519 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:37:21.508+08:00 and status AVAILABLE 2025-10-27 11:37:21.521 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:37:21.521 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:37:21.521 INFO 2458 --- [host-check-agent-status3,5344f8921b2aa0ed,9f3e334e1aa28914] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:37:21.522 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:37:21.514+08:00 and status AVAILABLE 2025-10-27 11:37:21.523 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:37:21.525 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:37:21.526 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:37:21.526 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:37:21.528 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:37:21.528 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:37:21.528 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:37:21.528 INFO 2458 --- [host-check-agent-status4,5344f8921b2aa0ed,70228f8b1b224d57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:37:21.528 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:37:21.529 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:37:21.528584729+08:00 2025-10-27 11:37:21.529 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:37:21.529+08:00 2025-10-27 11:37:21.532 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:37:21.535 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:37:21.535 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:37:21.535 INFO 2458 --- [host-check-agent-status2,5344f8921b2aa0ed,d9e17f00b33081c8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:37:21.538 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:37:21.529+08:00 and status AVAILABLE 2025-10-27 11:37:21.542 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:37:21.543 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:21.544 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:21.545 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:21.545 INFO 2458 --- [ocp-schedules-17,ed3aee5cb96aa31a,f88dee48427f309a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:37:21.546 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:21.548 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:21.548 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:21.549 INFO 2458 --- [ocp-schedules-3,876deee833c643c0,6c912d34babeb098] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:37:21.549 INFO 2458 --- [ocp-schedules-17,ed3aee5cb96aa31a,f88dee48427f309a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:21.551 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:21.554 INFO 2458 --- [ocp-schedules-5,400fd2f5e902ad42,88bc820e0e2edb9e] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:37:21.554 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:21.556 INFO 2458 --- [ocp-schedules-3,876deee833c643c0,6c912d34babeb098] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:21.557 INFO 2458 --- [ocp-schedules-3,876deee833c643c0,6c912d34babeb098] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:37:21.558 INFO 2458 --- [ocp-schedules-5,400fd2f5e902ad42,88bc820e0e2edb9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:21.565 INFO 2458 --- [ocp-schedules-22,b1e95c19e171b769,33f5f4d0e91343e5] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:37:21.569 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:21.572 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:21.597 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.598 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:21.599 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:37:21.606 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:21.612 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.615 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.616 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.617 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:21.618 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:21.626 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:21.627 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:37:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:37:21.628 INFO 2458 --- [ocp-schedules-17,ed3aee5cb96aa31a,f88dee48427f309a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:37:21.637 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:37:21.640 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:37:21.641 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:21.649 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:37:21.651 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.652 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:21.660 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:37:21.662 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:37:21.668 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:37:21.673 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:37:21.680 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:37:21.688 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:21.694 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.694 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.695 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:21.696 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:37:21.699 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:21.702 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:21.704 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:21.705 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.705 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:37:21.707 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:21.708 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:21.710 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:37:21.713 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:21.716 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:37:21.720 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:37:21.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:21.721528+08:00, lastProcessTime=2025-10-27T11:37:11.713278+08:00 2025-10-27 11:37:21.731 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:21.744 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:37:21.753 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:37:21.760 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:37:21.767 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:37:21.771 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:37:21.775 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:21.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.776 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:21.777 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.780 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:37:21.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:21.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.781 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.782 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:37:21.783 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:21.785 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:37:21.802 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:21.803 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:21.805 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:21.809 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:37:21.817 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:21.820 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:37:21.820 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:21.820 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:21.821 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:37:21.824 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:37:21.842 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:21.848 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:37:21.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:21.862 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:37:21.862 INFO 2458 --- [host-check-host-status5,7036af736834c4af,fb51fed549beb698] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:37:21.864 INFO 2458 --- [host-check-host-status4,7036af736834c4af,0e2ccb909b3f13cf] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:37:21.869 INFO 2458 --- [host-check-host-status2,7036af736834c4af,ab601b373585f264] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:37:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:21.885 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.888 INFO 2458 --- [host-check-host-status1,7036af736834c4af,c5f17f27db091942] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:37:21.888 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:21.888 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:37:21.889 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:21.890 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:21.890 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:21.891 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:37:21.898 INFO 2458 --- [host-check-host-status3,7036af736834c4af,cddc07ffc9d56ece] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:37:21.902 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:37:21.912 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:21.915 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:21.928 INFO 2458 --- [ocp-schedules-11,855abb6163e25c64,8a97e19e1b753250] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:37:21.930 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:21.930 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:21.970 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:37:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:22.014 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:37:22.014 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.015 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:37:22.016 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:21.721528+08:00 2025-10-27 11:37:22.017 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:22.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:22.018 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:37:22.034 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:37:22.055 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:22.055 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:22.060 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:22.060 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.061 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:37:22.087 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:37:22.087 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:37:22.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:22.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:22.129 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.171 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.172 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:22.176 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:37:22.186 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:37:22.196 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:22.199 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:22.247 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:37:22.295 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:37:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.295 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.296 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:37:22.297 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.297 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.298 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.298 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:37:22.333 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:22.373 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:22.376 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:37:22.378 INFO 2458 --- [host-check-clockdiff3,a318ba62e698f44a,60996c43e1ab7616] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536242 0 0 , error= 2025-10-27 11:37:22.378 INFO 2458 --- [host-check-clockdiff4,a318ba62e698f44a,09dc2099c3e3b3bf] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536242 0 0 , error= 2025-10-27 11:37:22.426 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:37:22.426 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:37:22.426 INFO 2458 --- [ocp-schedules-5,400fd2f5e902ad42,88bc820e0e2edb9e] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:37:22.676 WARN 2458 --- [ocp-async-8,7bb01f110eb4fd1b,3ca167067fcf3e0b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:22.678 INFO 2458 --- [ocp-async-3,8566c191ee98aadf,ddd440d136751e24] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:23.224 INFO 2458 --- [host-check-clockdiff5,a318ba62e698f44a,1b7a6cebcc045ffc] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536243 1 1 , error= 2025-10-27 11:37:23.707 INFO 2458 --- [prometheus-minute-schedule-1,c95516d03324954a,13f32a7161654770] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:23.714 INFO 2458 --- [prometheus-minute-schedule-1,c95516d03324954a,13f32a7161654770] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:24.677 WARN 2458 --- [ocp-async-5,a7c3312c61a24a95,716220a88a97d3a8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:24.678 INFO 2458 --- [ocp-async-4,d793f9e578523194,f4341c06f106bf5d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:24.708 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:24.709 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:26.019 INFO 2458 --- [ocp-schedules-11,46707363e98e5205,6172b61c0d50570e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:26.023 INFO 2458 --- [ocp-schedules-11,46707363e98e5205,6172b61c0d50570e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:26.677 WARN 2458 --- [ocp-async-9,b93bf85cca70bae4,4ef101b509556034] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:26.678 INFO 2458 --- [ocp-async-7,226bb5aeb8df5930,ee4ebc32dc0ad11f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:26.707 INFO 2458 --- [prometheus-minute-schedule-1,237c206430420ecd,2f64cf16193790a5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:26.714 INFO 2458 --- [prometheus-minute-schedule-1,237c206430420ecd,2f64cf16193790a5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:27.222 INFO 2458 --- [ocp-schedules-6,2cfbcb8087034103,c468d42f6e93e72b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:27.709 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:27.710 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:27.861 INFO 2458 --- [ocp-schedules-11,22c572efe81ab074,73cd86558ad0411e] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:37:27.869 INFO 2458 --- [ocp-schedules-11,22c572efe81ab074,73cd86558ad0411e] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:27.869 INFO 2458 --- [ocp-schedules-11,22c572efe81ab074,73cd86558ad0411e] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:28.544 INFO 2458 --- [ocp-async-10,67aaa07bd3fe2f07,ba2badfd24f82daf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:28.684 WARN 2458 --- [ocp-async-8,0ea96125dede8859,2b579e4e4e511a56] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:28.684 INFO 2458 --- [ocp-async-2,a8bf924c6bbfedeb,115700ba30348970] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:29.100 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:37:29.100 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:37:29.102 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:37:29.102 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839265989532524544 2025-10-27 11:37:29.103 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:37:29.103 INFO 2458 --- [ocp-schedules-19,fea64c1e54690172,60d1c027f1123e69] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:37:29.540 INFO 2458 --- [ocp-schedules-30,744483b419d25fc1,5a2753eb5c283fcd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:37:29.549 INFO 2458 --- [ocp-schedules-30,744483b419d25fc1,5a2753eb5c283fcd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:37:29.549 INFO 2458 --- [ocp-schedules-30,744483b419d25fc1,5a2753eb5c283fcd] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:37:29.705 INFO 2458 --- [ocp-schedules-21,6d7b6199b6605efe,5905103cd04f7a0f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:37:29.708 INFO 2458 --- [prometheus-minute-schedule-1,e4221f535e978fd5,4616de81948afcc7] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:29.716 INFO 2458 --- [prometheus-minute-schedule-1,e4221f535e978fd5,4616de81948afcc7] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:29.765 WARN 2458 --- [scanner-112220571-1,6d7b6199b6605efe,9767490d6afb133e] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:37:29.881 INFO 2458 --- [alarm-detect-1,6d7b6199b6605efe,e1afa226ef042118] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536220 ~ 1761536239 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536235, value=64.627)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536235, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536235=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:37:30.684 WARN 2458 --- [ocp-async-5,aea814dfd4d195bc,3e621bec7ed32373] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:30.684 INFO 2458 --- [ocp-async-6,b89325a5008be306,f9e2f2243f22eb35] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:30.710 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:30.712 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:31.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:31.714121+08:00, lastProcessTime=2025-10-27T11:37:21.721528+08:00 2025-10-27 11:37:31.915 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:31.714121+08:00 2025-10-27 11:37:32.684 INFO 2458 --- [ocp-async-9,a282f93cb5ec3996,da2e270967f8ebdf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:32.684 WARN 2458 --- [ocp-async-10,4322a1fa0f056106,1a6122a5274bb21d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:32.707 INFO 2458 --- [prometheus-minute-schedule-1,70ef364b3534ac01,b63354411d988704] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:32.713 INFO 2458 --- [prometheus-minute-schedule-1,70ef364b3534ac01,b63354411d988704] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:33.712 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:33.713 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:34.013 INFO 2458 --- [ocp-schedules-24,f5739da7593da363,566e25a2a04ebaaa] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:37:34.013 INFO 2458 --- [ocp-schedules-24,f5739da7593da363,566e25a2a04ebaaa] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:37:34.685 WARN 2458 --- [ocp-async-9,89e2d95cec6c2e3e,ea5f996a4ce6468f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:34.685 INFO 2458 --- [ocp-async-9,8a7ac70edee58f20,f807c622bf8170e7] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:35.707 INFO 2458 --- [prometheus-minute-schedule-1,bf676d1d3df988a0,4ef4d042c1125af1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:35.714 INFO 2458 --- [prometheus-minute-schedule-1,bf676d1d3df988a0,4ef4d042c1125af1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:36.032 INFO 2458 --- [ocp-schedules-2,216a231764b08234,1115a3ccacd791fc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:36.036 INFO 2458 --- [ocp-schedules-2,216a231764b08234,1115a3ccacd791fc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:36.685 WARN 2458 --- [ocp-async-9,419708371c9899cd,ba8c7eeac6ac0e57] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:36.685 INFO 2458 --- [ocp-async-2,72d8f0da8d82bc2d,3652716304e60b2c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:36.713 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:36.714 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:37.228 INFO 2458 --- [ocp-schedules-24,98562bd824cfae6b,94aa2c929c7465fa] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:37.708 INFO 2458 --- [prometheus-cache-monitor1,83e30e6abcf572e0,f83a22f22ba50534] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20320,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:37:37.708 INFO 2458 --- [prometheus-cache-monitor1,83e30e6abcf572e0,f83a22f22ba50534] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:37:38.544 INFO 2458 --- [ocp-async-9,6b301e54f6b5eea6,b34ec40aab85ff7f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:38.685 INFO 2458 --- [ocp-async-6,f2c5e3a9940e90c0,a81ef434c5eeee34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:38.685 WARN 2458 --- [ocp-async-10,f715055c0f4fda7a,227adcc8fef4e323] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:38.707 INFO 2458 --- [prometheus-minute-schedule-1,7ba3a62a7f70d085,d19429d533d360a9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:38.713 INFO 2458 --- [prometheus-minute-schedule-1,7ba3a62a7f70d085,d19429d533d360a9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:38.936 INFO 2458 --- [ocp-schedules-16,95a1eec4f0183781,b3b950f28646cb8b] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:37:38.943 INFO 2458 --- [ocp-schedules-16,95a1eec4f0183781,b3b950f28646cb8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:38.943 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:37:38.943 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:37:38.943 INFO 2458 --- [ocp-schedules-16,95a1eec4f0183781,b3b950f28646cb8b] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:37:39.714 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:39.715 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:39.888 INFO 2458 --- [ocp-schedules-9,c8a3b2a8228566b4,d9a967faab505c11] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:37:39.910 WARN 2458 --- [scanner-112220787-1,c8a3b2a8228566b4,a48c77a749ee1088] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:37:39.950 INFO 2458 --- [alarm-detect-2,c8a3b2a8228566b4,fe33e55c02e8c743] c.o.o.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, activeAt=2025-10-27T11:37:19.626+08:00 2025-10-27 11:37:40.010 INFO 2458 --- [ocp-schedules-14,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:37:40.010 INFO 2458 --- [ocp-schedules-5,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:37:40.034 INFO 2458 --- [alarm-detect-5,c8a3b2a8228566b4,7015da37b1fe88ba] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536235 ~ 1761536250 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536250, value=58.639)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536250, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536250=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:37:40.051 INFO 2458 --- [Check obproxy1,7712a74509560cad,6610db5119bac20b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:37:40.051 INFO 2458 --- [Check obproxy1,7712a74509560cad,6610db5119bac20b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:37:40.063 INFO 2458 --- [Collect obproxy1,a6390881e0a63703,4acd6d890c645c71] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:37:40.063 INFO 2458 --- [Collect obproxy1,a6390881e0a63703,4acd6d890c645c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:37:40.079 INFO 2458 --- [Check obproxy1,7712a74509560cad,6610db5119bac20b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:37:40.081 INFO 2458 --- [Check obproxy1,7712a74509560cad,6610db5119bac20b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:37:40.081 INFO 2458 --- [Check obproxy1,7712a74509560cad,6610db5119bac20b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:37:40.093 INFO 2458 --- [Collect obproxy1,a6390881e0a63703,4acd6d890c645c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:37:40.096 INFO 2458 --- [Collect obproxy1,a6390881e0a63703,4acd6d890c645c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:37:40.098 INFO 2458 --- [Collect obproxy1,a6390881e0a63703,4acd6d890c645c71] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:37:40.417 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:40.417 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:37:40.417 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:37:40.417 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:37:40.417 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:37:40.417 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:37:40.429 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:40.429 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:40.430 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:40.430 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:40.430 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:40.432 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:40.433 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:40.433 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:40.434 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:40.434 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:40.482 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:37:40.486 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:37:40.486 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:37:40.490 INFO 2458 --- [ForkJoinPool.commonPool-worker-5,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:37:40.499 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:37:40.502 INFO 2458 --- [ocp-schedules-21,e477ed190c5494d0,aebbd959b44c67c8] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:37:40.685 INFO 2458 --- [ocp-async-6,62e195c5907f93cf,560edd01cee872c2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:40.685 WARN 2458 --- [ocp-async-8,b43a6fb87a1f331a,96e4e157f2ab99fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:40.963 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:37:40.963 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:37:41.015 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=root 2025-10-27 11:37:41.015 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:41.015 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:41.056 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:41.056 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:41.057 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:41.058 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:37:41.069 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:41.069 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:41.070 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:41.070 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:37:41.099 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:37:41.121 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:41.122 INFO 2458 --- [ocp-schedules-1,e4511a772aa5e911,3de89473507c1a76] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:37:41.300 INFO 2458 --- [ocp-schedules-22,2b545990c6c87f5d,6c2960872466fd04] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:37:41.707 INFO 2458 --- [prometheus-minute-schedule-1,a3e98c4d9857978e,77b6d2b7fbd2675b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:41.713 INFO 2458 --- [prometheus-minute-schedule-1,a3e98c4d9857978e,77b6d2b7fbd2675b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:41.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:41.713313+08:00, lastProcessTime=2025-10-27T11:37:31.714121+08:00 2025-10-27 11:37:41.927 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:41.713313+08:00 2025-10-27 11:37:42.685 INFO 2458 --- [ocp-async-4,f2e4f69a44d46261,7ef1d52c53bf2459] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:42.685 WARN 2458 --- [ocp-async-6,be59f2a5f34569f3,75d70185ed198226] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:42.715 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:42.716 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:44.324 INFO 2458 --- [ocp-schedules-21,69e0506bbb763b12,184f3ae1f185a303] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:37:44.657 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1a38a8b751fef703,cfb42496df7e931c] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=1a38a8b751fef703, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:37:44.657 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,1a38a8b751fef703,cfb42496df7e931c] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=1a38a8b751fef703, duration=1 ms] 2025-10-27 11:37:44.685 INFO 2458 --- [ocp-async-4,a92f345669b8b486,b4302dc41c66acf8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:44.685 WARN 2458 --- [ocp-async-3,92c3a5a622c06008,8726232f969a817f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:44.707 INFO 2458 --- [prometheus-minute-schedule-1,5340d84e2b6104cb,f013a38a7d7f3b9c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:44.714 INFO 2458 --- [prometheus-minute-schedule-1,5340d84e2b6104cb,f013a38a7d7f3b9c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:45.102 INFO 2458 --- [prometheus-second-schedule-1,abaaa008403579fe,615c28fa30328f7b] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:37:45.716 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:45.718 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:46.046 INFO 2458 --- [ocp-schedules-22,7e8a90e0b28bc62b,2f679968c91f7442] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:46.050 INFO 2458 --- [ocp-schedules-22,7e8a90e0b28bc62b,2f679968c91f7442] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:46.686 INFO 2458 --- [ocp-async-4,3684251605d6edac,a2921a53518529fb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:46.686 WARN 2458 --- [ocp-async-7,78e52ca68a89cf95,68111c06f26ac59b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:47.235 INFO 2458 --- [ocp-schedules-10,498bdcd30b275bd3,b8f1f8f417d5ecca] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:47.707 INFO 2458 --- [prometheus-minute-schedule-1,4a462c4364263edb,b8b8d76fd7ea8831] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:47.714 INFO 2458 --- [prometheus-minute-schedule-1,4a462c4364263edb,b8b8d76fd7ea8831] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:47.871 INFO 2458 --- [ocp-schedules-15,d740855094a5d22b,1851e7723ba5d04d] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:37:47.880 INFO 2458 --- [ocp-schedules-15,d740855094a5d22b,1851e7723ba5d04d] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:47.880 INFO 2458 --- [ocp-schedules-15,d740855094a5d22b,1851e7723ba5d04d] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:37:48.251 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper 2025-10-27 11:37:48.275 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:48.280 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:48.332 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:48.335 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:48.386 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:48.389 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:48.438 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:48.441 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:48.501 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:48.504 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:48.544 INFO 2458 --- [ocp-async-8,d5b0d083521d8e72,e20aa055a53aaae5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:48.553 INFO 2458 --- [ocp-schedules-28,e01a811ee86a09a3,28f9f45bee87bc19] c.o.o.m.helper.ExporterRequestHelper : Refresh exporter helper finished, credentialSize=5, cachedSize=5 2025-10-27 11:37:48.686 INFO 2458 --- [ocp-async-6,fbbb078d9fc0b874,1fa83e883e94fd34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:48.686 WARN 2458 --- [ocp-async-9,12df72315404dace,eabeeb504e7da477] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:48.718 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:48.719 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:50.040 INFO 2458 --- [ocp-schedules-3,81fd7269df31bbfb,ddf668963185967d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:37:50.075 WARN 2458 --- [scanner-112220998-1,81fd7269df31bbfb,440ea03fa533fbae] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:37:50.116 INFO 2458 --- [alarm-detect-3,81fd7269df31bbfb,6df561ecfa841d0f] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536200 ~ 1761536260 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536220, value=204554.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204554.56, values={1761536220=204554.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:37:50.161 INFO 2458 --- [alarm-detect-15,81fd7269df31bbfb,d31b8196f1517de3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536200 ~ 1761536260 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536220, value=93.251)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.251, values={1761536220=93.251}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:37:50.177 INFO 2458 --- [alarm-detect-8,81fd7269df31bbfb,a31593197ae58956] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536200 ~ 1761536260 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536220, value=204555.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204555.912, values={1761536220=204555.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:37:50.205 INFO 2458 --- [ocp-schedules-8,06e30559e5c747cd,8a6f08def6ee351a] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:37:50.205+08:00, clusters=[1, 2] 2025-10-27 11:37:50.221 INFO 2458 --- [ocp-schedules-8,06e30559e5c747cd,8a6f08def6ee351a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:37:50.224 INFO 2458 --- [ocp-schedules-8,06e30559e5c747cd,8a6f08def6ee351a] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:37:50.686 WARN 2458 --- [ocp-async-10,86329fc4ce5f74cd,c094312de0f36e7f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:50.686 INFO 2458 --- [ocp-async-6,312e09667e6a5987,4b5d72429e512e32] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:50.707 INFO 2458 --- [prometheus-minute-schedule-1,95dae2e3046df118,6a0c3e6c8449f7b9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:50.713 INFO 2458 --- [prometheus-minute-schedule-1,95dae2e3046df118,6a0c3e6c8449f7b9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:51.408 INFO 2458 --- [ocp-schedules-25,ded2ab30483ee2e9,a67f81aee5e928b5] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:51.409 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:37:51.409 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:37:51.409 INFO 2458 --- [ocp-schedules-21,0bdf37420c27973a,6603c09e73d5218a] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:51.409 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:37:51.410 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:37:51.410 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:37:51.410 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:37:51.410 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:37:51.413 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:37:51.416 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:37:51.416 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:37:51.421 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:51.422 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:51.422 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:51.423 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:51.426 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:51.427 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:51.427 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:51.428 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:51.428 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:51.428 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:51.428 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:51.429 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:51.430 INFO 2458 --- [ocp-schedules-11,0a617b985eb897fc,28bd756853413f51] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:37:51.430 INFO 2458 --- [host-check-clockdiff1,0a617b985eb897fc,2f347cec68c19547] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-clockdiff4,0a617b985eb897fc,cc12fca0098a158f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-clockdiff3,0a617b985eb897fc,5f02af741a972d27] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-clockdiff2,0a617b985eb897fc,6ead28ff4f545b7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:37:51.431 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:51.432 INFO 2458 --- [host-check-clockdiff5,0a617b985eb897fc,6fc42ebf163b2754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:37:51.441 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:51.442 INFO 2458 --- [ocp-schedules-29,9404f68acf0d4ba9,101b8dcc840eaa5f] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:37:51.442 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:51.445 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:51.449 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:51.471 INFO 2458 --- [host-check-clockdiff2,0a617b985eb897fc,6ead28ff4f545b7e] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536271 -1 -1 , error= 2025-10-27 11:37:51.475 INFO 2458 --- [host-check-clockdiff3,0a617b985eb897fc,5f02af741a972d27] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536271 -1 -1 , error= 2025-10-27 11:37:51.475 INFO 2458 --- [host-check-clockdiff1,0a617b985eb897fc,2f347cec68c19547] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536271 0 0 , error= 2025-10-27 11:37:51.478 INFO 2458 --- [host-check-clockdiff4,0a617b985eb897fc,cc12fca0098a158f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536271 0 0 , error= 2025-10-27 11:37:51.531 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:37:51.534 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:37:51.533476875+08:00 2025-10-27 11:37:51.534 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:37:51.534+08:00 2025-10-27 11:37:51.535 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:37:51.536 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:37:51.539 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:37:51.540 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:37:51.539207762+08:00 2025-10-27 11:37:51.541 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:37:51.541 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:37:51.540+08:00 2025-10-27 11:37:51.542 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:37:51.542 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:37:51.542 INFO 2458 --- [host-check-agent-status2,0bdf37420c27973a,c1593b539e440735] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:37:51.543 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:37:51.544 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:37:51.544 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:37:51.544 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:37:51.544 INFO 2458 --- [host-check-agent-status1,0bdf37420c27973a,9683153c10c667dc] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:37:51.544 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:37:51.544918801+08:00 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:37:51.544403987+08:00 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-agent-status5,0bdf37420c27973a,860b43e00019bd5b] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:37:51.545 INFO 2458 --- [ocp-schedules-9,1a857a8f7e446e31,e68f72d441030fd6] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:37:51.545+08:00 2025-10-27 11:37:51.545 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:37:51.534+08:00 and status AVAILABLE 2025-10-27 11:37:51.546 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:37:51.545+08:00 2025-10-27 11:37:51.546 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:37:51.546 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:37:51.546 INFO 2458 --- [host-check-agent-status3,0bdf37420c27973a,9cf2e6c966f026d8] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:37:51.547 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:37:51.548 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:37:51.548 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:37:51.548678530+08:00 2025-10-27 11:37:51.549 INFO 2458 --- [ocp-schedules-1,dcd207c173491f6a,87ffd0588c0fc05f] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:37:51.549 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:37:51.548+08:00 2025-10-27 11:37:51.549 INFO 2458 --- [ocp-schedules-9,1a857a8f7e446e31,e68f72d441030fd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:51.550 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:37:51.550 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:37:51.550 INFO 2458 --- [host-check-agent-status4,0bdf37420c27973a,c76f2bdade442036] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:37:51.551 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:37:51.552 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:37:51.540+08:00 and status AVAILABLE 2025-10-27 11:37:51.553 INFO 2458 --- [ocp-schedules-1,dcd207c173491f6a,87ffd0588c0fc05f] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:51.554 INFO 2458 --- [ocp-schedules-3,c851f52c8845f4a6,236eb0210d1c37da] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:37:51.554 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:37:51.545+08:00 and status AVAILABLE 2025-10-27 11:37:51.554 INFO 2458 --- [ocp-schedules-1,dcd207c173491f6a,87ffd0588c0fc05f] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:37:51.555 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:37:51.545+08:00 and status AVAILABLE 2025-10-27 11:37:51.557 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:37:51.558 INFO 2458 --- [ocp-schedules-3,c851f52c8845f4a6,236eb0210d1c37da] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:51.558 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:37:51.559 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:37:51.548+08:00 and status AVAILABLE 2025-10-27 11:37:51.560 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:37:51.564 INFO 2458 --- [ocp-schedules-4,2ba56e4bf4319b64,1f7cc9317e5e6eea] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:37:51.564 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:37:51.568 INFO 2458 --- [ocp-schedules-24,b6de63219d84fa9b,dc2c715e84166f09] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:37:51.570 INFO 2458 --- [ocp-schedules-22,822f358f1dcd12a7,aa079efcbb8470d7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:37:51.572 INFO 2458 --- [ocp-schedules-20,aded5a668084a00e,de48eb56879e7e21] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster root server. 2025-10-27 11:37:51.582 INFO 2458 --- [ocp-schedules-20,aded5a668084a00e,de48eb56879e7e21] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:51.582 INFO 2458 --- [ocp-schedules-22,822f358f1dcd12a7,aa079efcbb8470d7] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:37:51.583 INFO 2458 --- [ocp-schedules-24,b6de63219d84fa9b,dc2c715e84166f09] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:37:51.586 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:37:51.589 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:37:51.593 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:51.596 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:51.596 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:51.599 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:51.599 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:37:51.602 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:37:51.604 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:37:51.607 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:37:51.616 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.617 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:51.618 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:37:51.628 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.628 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:37:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.629 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:51.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:37:51.634 INFO 2458 --- [ocp-schedules-9,1a857a8f7e446e31,e68f72d441030fd6] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:37:51.636 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,0ea56d135ccaf51b,8d76953e237dcc38] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase2&ObClusterId=1760510188&version=2, client=10.40.65.146, traceId=0ea56d135ccaf51b, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase2,1760510188,false,2,] 2025-10-27 11:37:51.636 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,0ea56d135ccaf51b,8d76953e237dcc38] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=0ea56d135ccaf51b, duration=1 ms] 2025-10-27 11:37:51.637 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.638 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:37:51.644 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.648 INFO 2458 --- [Check root service 2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.66.107, 2882] 2025-10-27 11:37:51.658 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:51.661 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.662 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.662 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.662 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.662 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.663 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [obconfig_url] 2025-10-27 11:37:51.668 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:51.670 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:51.670 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.672 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:51.673 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:37:51.674 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:51.680 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:37:51.693 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:37:51.694 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:37:51.694 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cb9948705a8b4e63,27bd48b72b78af83] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&ObClusterId=1760510187&version=2, client=10.40.65.146, traceId=cb9948705a8b4e63, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getRootServiceInfo(String,Long,String,Long,boolean,Integer), args=myoceanbase,1760510187,false,2,] 2025-10-27 11:37:51.694 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-4,cb9948705a8b4e63,27bd48b72b78af83] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.65.146, traceId=cb9948705a8b4e63, duration=1 ms] 2025-10-27 11:37:51.696 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.696 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.697 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.697 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [rootservice_list] 2025-10-27 11:37:51.701 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:37:51.703 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:37:51.716 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.716 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.717 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.718 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:37:51.713199+08:00, lastProcessTime=2025-10-27T11:37:41.713313+08:00 2025-10-27 11:37:51.719 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:51.719 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.146, 2882] 2025-10-27 11:37:51.720 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:51.720 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.720 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.720 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.724 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.65.168, 2882] 2025-10-27 11:37:51.725 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.725 INFO 2458 --- [Check root service 1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.726 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.728 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.728 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.730 INFO 2458 --- [Check root service 1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port AS inner_port, with_rootserver, UPPER(`status`) AS `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 AS stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 AS start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 AS block_migrate_in_time FROM DBA_OB_SERVERS WHERE svr_ip = ? AND svr_port = ?, args: [10.40.67.22, 2882] 2025-10-27 11:37:51.730 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.730 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.730 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.731 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:51.731 INFO 2458 --- [ocp-schedules-20,aded5a668084a00e,de48eb56879e7e21] .o.o.s.t.b.c.AbstractShardingClusterTask : Check root service executed completed, allCount=2, successCount=2 2025-10-27 11:37:51.736 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:37:51.739 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:37:51.740 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:37:51.747 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:37:51.750 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:37:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:51.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:37:51.773 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:51.773 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:51.777 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:51.778 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:51.792 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:37:51.793 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:37:51.798 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:37:51.803 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:37:51.804 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:37:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.814 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.815 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:37:51.821 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:37:51.827 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:51.830 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.831 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:37:51.833 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:37:51.834 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:51.837 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:37:51.837 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:37:51.839 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:37:51.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:51.854 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:51.866 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:51.869 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:51.869 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:37:51.875 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:51.877 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:51.879 INFO 2458 --- [ocp-schedules-13,0b2034d56f6b013a,37b9bc266d0005ab] c.o.o.s.common.scheduled.OcpSchedules : Cleanup ocp connection pools. 2025-10-27 11:37:51.879 INFO 2458 --- [ocp-schedules-13,0b2034d56f6b013a,37b9bc266d0005ab] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] clean up the expired connectors. 2025-10-27 11:37:51.881 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:37:51.883 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:51.889 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:37:51.897 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:37:51.901 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.903 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.904 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:37:51.905 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:51.905 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:51.906 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.906 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:37:51.909 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:37:51.909 INFO 2458 --- [host-check-host-status4,ded2ab30483ee2e9,38ac32f519028e04] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:37:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:51.914 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:51.915 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:51.917 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:37:51.917 INFO 2458 --- [host-check-host-status3,ded2ab30483ee2e9,8c44ceef7c489e12] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:37:51.918 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:37:51.930 INFO 2458 --- [host-check-host-status2,ded2ab30483ee2e9,be6cc3586f5cb0b6] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:37:51.931 INFO 2458 --- [ocp-schedules-14,a64f128365d54fd1,7b0e3f4462f1ea31] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:37:51.935 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:37:51.939 INFO 2458 --- [host-check-host-status1,ded2ab30483ee2e9,8dfaba899f791e8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:37:51.944 INFO 2458 --- [host-check-host-status5,ded2ab30483ee2e9,17becda43fa60a23] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:37:51.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:37:51.948 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:37:51.952 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:51.952 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:51.979 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:37:51.713199+08:00 2025-10-27 11:37:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:51.992 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:52.000 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:37:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:52.024 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:52.043 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:37:52.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.044 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:37:52.046 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.047 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:37:52.062 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:37:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:52.064 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:37:52.087 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:52.087 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.088 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:37:52.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:52.099 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:37:52.110 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:37:52.110 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:37:52.123 INFO 2458 --- [pool-overview-container-3,ac6780ebb3cb6e33,69f593a3f1bcd88a] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.136 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.173 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.174 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:37:52.183 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:37:52.193 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:37:52.204 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:37:52.206 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:37:52.249 INFO 2458 --- [pool-overview-container-1,4db4249b5a6e17dd,8eb5729a1b0eae62] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.267 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:37:52.289 INFO 2458 --- [pool-overview-container-6,930b09228060fe97,a86907f165b946b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.291 INFO 2458 --- [pool-overview-container-6,930b09228060fe97,a86907f165b946b9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ SVR_IP, SVR_PORT, ZONE, CPU_CAPACITY, CPU_CAPACITY_MAX, CPU_ASSIGNED, CPU_ASSIGNED_MAX, MEM_CAPACITY, MEM_CAPACITY AS MEM_CAPACITY_MAX, MEM_ASSIGNED, MEM_ASSIGNED AS MEM_ASSIGNED_MAX, LOG_DISK_CAPACITY, LOG_DISK_ASSIGNED, LOG_DISK_IN_USE, DATA_DISK_CAPACITY, DATA_DISK_IN_USE FROM oceanbase.GV$OB_SERVERS 2025-10-27 11:37:52.317 INFO 2458 --- [pool-12-thread-9,4db4249b5a6e17dd,157e29940a4f233d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to get cluster trans stat 2025-10-27 11:37:52.319 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:37:52.319 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.319 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.320 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.320 INFO 2458 --- [pool-12-thread-12,4db4249b5a6e17dd,c037106afd3fc057] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get outline history 2025-10-27 11:37:52.320 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:37:52.322 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.322 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.322 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.323 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:37:52.324 INFO 2458 --- [pool-12-thread-11,4db4249b5a6e17dd,b7942fcc0eea4024] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get Cluster Sql Stat 2025-10-27 11:37:52.324 INFO 2458 --- [pool-12-thread-11,4db4249b5a6e17dd,b7942fcc0eea4024] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster slow Sql Stat 2025-10-27 11:37:52.327 INFO 2458 --- [ForkJoinPool-7-worker-14,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.328 INFO 2458 --- [ForkJoinPool-7-worker-7,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.330 INFO 2458 --- [pool-12-thread-10,4db4249b5a6e17dd,c8723d7b9a65401e] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster session stat 2025-10-27 11:37:52.338 INFO 2458 --- [pool-12-thread-9,4db4249b5a6e17dd,157e29940a4f233d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : begin to attach trans id 2025-10-27 11:37:52.338 INFO 2458 --- [pool-12-thread-9,4db4249b5a6e17dd,157e29940a4f233d] c.o.o.p.t.i.ObTransactionDiagServiceImpl : finish fetching trans data 2025-10-27 11:37:52.338 INFO 2458 --- [ForkJoinPool-7-worker-9,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.339 INFO 2458 --- [ForkJoinPool-7-worker-0,,] c.o.ocp.service.iam.user.UserService : user 100 login with organization 10000000 2025-10-27 11:37:52.339 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase2 2025-10-27 11:37:52.339 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase2 2025-10-27 11:37:52.340 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get tenant session stat of myoceanbase 2025-10-27 11:37:52.340 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get session of myoceanbase 2025-10-27 11:37:52.363 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:37:52.367 INFO 2458 --- [pool-12-thread-11,4db4249b5a6e17dd,b7942fcc0eea4024] c.o.o.p.oas.internal.OasStatServiceImpl : begin to get cluster anomaly sql Stat 2025-10-27 11:37:52.373 INFO 2458 --- [pool-12-thread-11,4db4249b5a6e17dd,b7942fcc0eea4024] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching cluster sql stat 2025-10-27 11:37:52.396 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.398 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:37:52.404 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.405 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:37:52.405 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.405 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.406 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.406 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.406 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.406 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.406 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:37:52.408 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:37:52.414 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:37:52.418 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.418 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.418 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.418 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.419 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.419 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:37:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.423 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.424 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.424 INFO 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:37:52.425 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:37:52.425 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.427 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:37:52.427 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.428 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:37:52.445 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.445 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? AND state = 'ACTIVE' 2025-10-27 11:37:52.446 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.446 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(*) FROM GV$OB_PROCESSLIST WHERE 1=1 AND tenant = ? 2025-10-27 11:37:52.447 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.p.o.i.OasSessionStatServiceImpl : begin to get deadlock of myoceanbase2 2025-10-27 11:37:52.457 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=test, dbUser=root 2025-10-27 11:37:52.457 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.457 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.458 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.458 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.458 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.459 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.459 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:37:52.460 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.460 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=test, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.460 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.461 INFO 2458 --- [ForkJoinPool-7-worker-7,4db4249b5a6e17dd,259b7da5048aa409] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:37:52.463 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1002] 2025-10-27 11:37:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:37:52.499 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:37:52.499 INFO 2458 --- [ocp-schedules-3,c851f52c8845f4a6,236eb0210d1c37da] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:37:52.506 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=test2, dbUser=root 2025-10-27 11:37:52.508 WARN 2458 --- [ForkJoinPool-7-worker-9,4db4249b5a6e17dd,2b6613d26365c3eb] c.o.o.p.oas.internal.OasStatServiceImpl : Failed to get session info of :myoceanbase com.oceanbase.ocp.service.profile.exception.CredentialMissingException: [OCP CredentialMissingException]: status=428 PRECONDITION_REQUIRED, errorCode=IAM_CREDENTIAL_REQUIRED, args=myoceanbase,ocp_monitor,root at com.oceanbase.ocp.service.profile.exception.CredentialMissingException.ob(CredentialMissingException.java:34) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireJdbcPasswordInner(CredentialServiceImpl.java:167) at com.oceanbase.ocp.service.credential.CredentialServiceImpl.acquireObPassword(CredentialServiceImpl.java:144) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$FastClassBySpringCGLIB$$6855c59e.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.service.credential.CredentialServiceImpl$$EnhancerBySpringCGLIB$$3d3c7d57.acquireObPassword() at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.acquirePassword(ConnectPropertiesBuilder.java:204) at com.oceanbase.ocp.service.ob.obsdk.factory.ConnectPropertiesBuilder.buildConnectProperties(ConnectPropertiesBuilder.java:97) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.buildConnectProperties(ObAccessorFactory.java:180) at com.oceanbase.ocp.service.ob.obsdk.factory.ObAccessorFactory.createObAccessor(ObAccessorFactory.java:57) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.lambda$countSession$1(TenantSessionServiceImpl.java:243) at java.lang.Iterable.forEach(Iterable.java:75) at com.oceanbase.ocp.obops.internal.tenant.TenantSessionServiceImpl.countSession(TenantSessionServiceImpl.java:215) at com.oceanbase.ocp.perf.oas.internal.OasSessionStatServiceImpl.getTenantSessionStat(OasSessionStatServiceImpl.java:71) at com.oceanbase.ocp.perf.oas.internal.OasStatServiceImpl.lambda$null$8(OasStatServiceImpl.java:281) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:290) at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731) at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175) 2025-10-27 11:37:52.515 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.517 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:37:52, 2025-10-27 11:37:52] 2025-10-27 11:37:52.517 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.519 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, case enabled when 'YES' then 1 else 0 end as enabled, create_time as gmt_create from DBA_OB_OUTLINE_CONCURRENT_HISTORY where ADDDATE(create_time, ?) > CURDATE() and is_deleted = 'NO' 2025-10-27 11:37:52.537 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.537 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.537 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.537 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.537 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.542 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(60000000) ocp_query */ time_to_usec(t1.MODIFY_TIME) AS UPDATE_TIME, t1.RESOURCE_POOL_ID, t1.NAME, t1.UNIT_COUNT, t1.UNIT_CONFIG_ID, t1.ZONE_LIST, t1.TENANT_ID, t1.REPLICA_TYPE, t2.NAME AS UNIT_CONFIG_NAME, t2.MAX_CPU, t2.MIN_CPU, t2.MEMORY_SIZE AS MAX_MEMORY, t2.MEMORY_SIZE AS MIN_MEMORY, t2.LOG_DISK_SIZE, t2.MAX_IOPS, t2.MIN_IOPS, t2.IOPS_WEIGHT FROM oceanbase.DBA_OB_RESOURCE_POOLS AS t1 JOIN oceanbase.DBA_OB_UNIT_CONFIGS AS t2 ON t1.UNIT_CONFIG_ID = t2.UNIT_CONFIG_ID AND t1.tenant_id = ?, args: [1006] 2025-10-27 11:37:52.543 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:37:52.544 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.546 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ event_id, svr_ip, svr_port, report_time, cycle_idx as cycle_index, cycle_size, role, visitor, object as queryResource, extra_name1, extra_value1 from DBA_OB_DEADLOCK_EVENT_HISTORY where report_time > ? and report_time < ?;, args: [2025-10-20 11:37:52, 2025-10-27 11:37:52] 2025-10-27 11:37:52.548 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:37:52.548 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.p.o.i.OasSessionStatServiceImpl : finish fetching tenant session data of myoceanbase2 2025-10-27 11:37:52.548 INFO 2458 --- [ForkJoinPool-7-worker-0,4db4249b5a6e17dd,5bb3bb74d9ea6100] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching tenant session stat of myoceanbase2 2025-10-27 11:37:52.548 INFO 2458 --- [pool-12-thread-10,4db4249b5a6e17dd,c8723d7b9a65401e] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching session stat 2025-10-27 11:37:52.548 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.549 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ outline_name as outlineName,outline_id, outline_content, database_id, database_name, visible_signature, sql_text, concurrent_num, limit_target, create_time as gmt_create, modify_time from DBA_OB_CONCURRENT_LIMIT_SQL 2025-10-27 11:37:52.550 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.551 INFO 2458 --- [ForkJoinPool-7-worker-14,4db4249b5a6e17dd,62e5eb42d9ddbc22] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ database_id, tenant_id, outline_id, outline_name as outlineName, outline_content, sql_text, sql_id, outline_target, create_time as gmt_create, modify_time from DBA_OB_OUTLINES 2025-10-27 11:37:52.554 INFO 2458 --- [pool-12-thread-12,4db4249b5a6e17dd,c037106afd3fc057] c.o.o.p.oas.internal.OasStatServiceImpl : finish fetching outline history 2025-10-27 11:37:52.573 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:52.578 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.582 INFO 2458 --- [refresh-tenant-resource-cache-2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1002] 2025-10-27 11:37:52.663 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:37:52.663 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.663 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.663 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:37:52.663 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:37:52.664 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:37:52.668 INFO 2458 --- [refresh-tenant-resource-cache-4,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ QUERY_TIMEOUT(5000000) ocp_query */ t1.UNIT_ID, t1.RESOURCE_POOL_ID, t1.UNIT_CONFIG_ID, t1.ZONE, t1.SVR_IP, t1.SVR_PORT, t1.STATUS, t2.REPLICA_TYPE, t2.TENANT_ID, t3.TENANT_NAME, t2.NAME AS RESOURCE_POOL_NAME, t2.MODIFY_TIME AS RESOURCE_POOL_UPDATE_TIME, t1.MIGRATE_FROM_SVR_IP, t1.MIGRATE_FROM_SVR_PORT, t1.MANUAL_MIGRATE FROM oceanbase.DBA_OB_UNITS AS t1 JOIN oceanbase.DBA_OB_RESOURCE_POOLS AS t2 ON t1.RESOURCE_POOL_ID = t2.RESOURCE_POOL_ID JOIN oceanbase.DBA_OB_TENANTS t3 ON t2.TENANT_ID = t3.TENANT_ID WHERE 1=1 AND t2.tenant_id = ?, args: [1006] 2025-10-27 11:37:52.686 WARN 2458 --- [ocp-async-9,8acd895dd410a9da,e791d5d2cac61369] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:52.686 INFO 2458 --- [ocp-async-8,4c007a790ca1b1bc,88c8645d8b08af94] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:53.164 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:37:53.221 INFO 2458 --- [host-check-clockdiff5,0a617b985eb897fc,6fc42ebf163b2754] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536273 1 1 , error= 2025-10-27 11:37:53.237 INFO 2458 --- [ocp-schedules-14,bfc31a99695884df,6b15472dd0b1a914] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20320,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:37:53.237 INFO 2458 --- [ocp-schedules-14,bfc31a99695884df,6b15472dd0b1a914] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:37:53.707 INFO 2458 --- [prometheus-minute-schedule-1,abb149f979da6ded,ce63bd3d7a1cd160] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:53.714 INFO 2458 --- [prometheus-minute-schedule-1,abb149f979da6ded,ce63bd3d7a1cd160] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:54.272 INFO 2458 --- [ocp-updater-0,,] c.o.ocp.obsdk.connector.ObVersionHolder : Invalid expired ob version cache 2025-10-27 11:37:54.562 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting SQL_PERF_DIAGNOSER, Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:54.570 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER : for [Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandb 2025-10-27 11:37:54.586 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:54.686 INFO 2458 --- [ocp-async-2,12197e9e5867d545,32dd53e1bec6ad46] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:54.686 WARN 2458 --- [ocp-async-10,e8208cea4c65abd4,abd8777fda190004] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:54.689 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:54.701 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:54.712 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:54.720 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:54.721 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:55.032 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.038 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.038 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:37:55.041 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.046 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.053 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.055 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.626 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.628 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.629 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:55.848 INFO 2458 --- [ocp-schedules-7,82b1a5f55427773e,281b98aa93bae294] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:37:56.064 INFO 2458 --- [ocp-schedules-15,e2dad06d13113ced,2c22df58f8a77ba1] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:37:56.068 INFO 2458 --- [ocp-schedules-15,e2dad06d13113ced,2c22df58f8a77ba1] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:37:56.093 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.094 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.096 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.097 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.098 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.099 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.100 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.110 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.111 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.112 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.113 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.114 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : ROW_LOCK_CONTENTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.114 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : PERFORMANCE_DEGRADATION is disabled or lacking of config 2025-10-27 11:37:56.115 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : EXECUTION_SPIKE found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.116 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : CPU_TIME_PROPORTION_HIGH found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.117 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.SqlPerfDiagnoseService : DML_WITH_UNNECESSARY_PARTITION found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:31:54.167, end=2025-10-27T11:36:54.557} 2025-10-27 11:37:56.117 INFO 2458 --- [pool-19-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : SQL_PERF_DIAGNOSER, Found out 0 dubious sqls 2025-10-27 11:37:56.686 INFO 2458 --- [ocp-async-1,ab988426dc969e7f,74b6131f38d25fae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:56.686 WARN 2458 --- [ocp-async-6,07eecae54797b86c,2dff368f44bc7481] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:56.707 INFO 2458 --- [prometheus-minute-schedule-1,2189f5a52a49c0ac,c975f9d61b087894] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:56.718 INFO 2458 --- [prometheus-minute-schedule-1,2189f5a52a49c0ac,c975f9d61b087894] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:37:57.242 INFO 2458 --- [ocp-schedules-18,0ccd4f2a314bdf18,155fe7fd04914282] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:37:57.721 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:37:57.722 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:37:58.505 INFO 2458 --- [ocp-schedules-28,1fd0630487c5d804,b8fb974fb4640dfd] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:37:58.509 INFO 2458 --- [ocp-schedules-28,1fd0630487c5d804,b8fb974fb4640dfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:37:58.544 INFO 2458 --- [ocp-async-1,d96c0791d6cb25f3,47296cf37f2713de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:37:58.565 INFO 2458 --- [ocp-schedules-28,1fd0630487c5d804,b8fb974fb4640dfd] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:37:58.686 WARN 2458 --- [ocp-async-9,acc476771e57221e,91a5a5a1ad5d38e4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:37:58.687 INFO 2458 --- [ocp-async-4,c813b558d586f721,d162cda791c09b57] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:37:59.603 INFO 2458 --- [ocp-schedules-24,8ca93f991854b58e,b591019efeb25475] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:37:59.707 INFO 2458 --- [prometheus-minute-schedule-1,d053a77829bf54c8,59115315f306990f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536280, interval=60 2025-10-27 11:37:59.717 INFO 2458 --- [prometheus-minute-schedule-1,d053a77829bf54c8,59115315f306990f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:00.210 INFO 2458 --- [ocp-schedules-28,882db92960b11920,89263a50e93953c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:00.219 WARN 2458 --- [scanner-112221213-1,882db92960b11920,8cdc658f5fb2e718] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:00.283 INFO 2458 --- [alarm-detect-1,882db92960b11920,60a4ff53bb2b7367] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536250 ~ 1761536270 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536265, value=1.513)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.513, values={1761536265=1.513}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:00.283 INFO 2458 --- [alarm-detect-1,882db92960b11920,60a4ff53bb2b7367] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:38:00.360 INFO 2458 --- [alarm-detect-12,882db92960b11920,d238fd842dff6fbf] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536250 ~ 1761536270 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536265, value=51.143)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536265, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536265=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:00.687 WARN 2458 --- [ocp-async-3,b0183f397ad7d55b,4872ac5874b996c3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:00.687 INFO 2458 --- [ocp-async-2,0c3a47bbb1da028e,ebb1e3c0b81f4385] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:00.722 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:00.723 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:01.503 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,24243667c54aa6ba,425cce61cabad80d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=24243667c54aa6ba, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:38:01.504 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,24243667c54aa6ba,425cce61cabad80d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=24243667c54aa6ba, duration=2 ms] 2025-10-27 11:38:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:01.715417+08:00, lastProcessTime=2025-10-27T11:37:51.713199+08:00 2025-10-27 11:38:02.275 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:01.715417+08:00 2025-10-27 11:38:02.687 WARN 2458 --- [ocp-async-1,02f239700396de16,d32896c5b3828243] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:02.687 INFO 2458 --- [ocp-async-7,aa66adaeda183f99,1165476878211a4e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:02.707 INFO 2458 --- [prometheus-minute-schedule-1,bc2788890c219668,bea1466f25ec2f3a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:02.713 INFO 2458 --- [prometheus-minute-schedule-1,bc2788890c219668,bea1466f25ec2f3a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:03.723 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:03.724 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:04.687 WARN 2458 --- [ocp-async-8,c86f30824432b50a,7e694bc1c51a5c54] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:04.687 INFO 2458 --- [ocp-async-8,013a5109934090e4,d749bdd3c2a5fa42] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:05.065 INFO 2458 --- [ocp-schedules-10,8688fbb6536235a7,e341ecebd6a7aca0] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:38:05.073 INFO 2458 --- [ocp-schedules-10,8688fbb6536235a7,e341ecebd6a7aca0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:05.114 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:05.114 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:38:05.129 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:05.129 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:05.129 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:05.129 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:05.130 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:05.130 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:38:05.144 INFO 2458 --- [ocp-schedules-10,8688fbb6536235a7,e341ecebd6a7aca0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:38:05.707 INFO 2458 --- [prometheus-minute-schedule-1,edcf95220f2222a8,d4dcbf662907e228] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:05.714 INFO 2458 --- [prometheus-minute-schedule-1,edcf95220f2222a8,d4dcbf662907e228] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:05.998 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:38:06.055 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:38:06.076 INFO 2458 --- [ocp-schedules-16,b7e0e418e681df2f,b40d2b8dd1b96b82] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:06.079 INFO 2458 --- [ocp-schedules-16,b7e0e418e681df2f,b40d2b8dd1b96b82] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:06.101 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:38:06.124 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:38:06.124 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:38:06.124 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:38:06.124 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:38:06.126 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:38:06.126 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:38:06.128 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:38:06.128 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:38:06.128 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:38:06.128 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:38:06.129 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:06.130 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:38:06.131 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:06.132 INFO 2458 --- [sql-rollup-worker-0,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:38:06.196 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:35:30, start=2025-10-27T11:32:30, end=2025-10-27T11:35:00, updates=221, granule=120000ms, delay=180000ms 2025-10-27 11:38:06.203 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:35:30, target=2025-10-27T11:35:00, granule=120000ms, delay=30000ms) 2025-10-27 11:38:06.203 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-27T11:35:00, target=2025-10-27T11:22:30, granule=600000ms, delay=750000ms) 2025-10-27 11:38:06.203 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=205ms, period=120000ms 2025-10-27 11:38:06.687 WARN 2458 --- [ocp-async-3,81880585844b8107,41ded515f80f3b02] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:06.687 INFO 2458 --- [ocp-async-10,c8cf1d50677a0987,9e997647581e696a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:06.724 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:06.726 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:07.250 INFO 2458 --- [ocp-schedules-27,895e18c443f47f24,031b1c287ea0e608] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:07.882 INFO 2458 --- [ocp-schedules-14,77add2fad53cb898,6496053ea64d8783] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:38:07.891 INFO 2458 --- [ocp-schedules-14,77add2fad53cb898,6496053ea64d8783] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:07.891 INFO 2458 --- [ocp-schedules-14,77add2fad53cb898,6496053ea64d8783] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:08.544 INFO 2458 --- [ocp-async-2,7f0273530870191f,d886f97f08c3c037] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:08.687 INFO 2458 --- [ocp-async-6,ac15ee59edcf877c,6150a29484aff169] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:08.687 WARN 2458 --- [ocp-async-5,b5938b1c25f64bb5,515fe1393519de4e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:08.707 INFO 2458 --- [prometheus-minute-schedule-1,e8b45d3ef9046cbd,6e1f53aa801e5cdf] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:08.713 INFO 2458 --- [prometheus-minute-schedule-1,e8b45d3ef9046cbd,6e1f53aa801e5cdf] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:09.726 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:09.727 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:10.008 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:38:10.008 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:38:10.045 INFO 2458 --- [Check obproxy1,9db67cc52605f225,cf00c0855c5247c5] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:38:10.045 INFO 2458 --- [Check obproxy1,9db67cc52605f225,cf00c0855c5247c5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:38:10.051 INFO 2458 --- [Collect obproxy1,55938d299aed835a,40a3b6636bb4fca6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:38:10.052 INFO 2458 --- [Collect obproxy1,55938d299aed835a,40a3b6636bb4fca6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:38:10.075 INFO 2458 --- [Check obproxy1,9db67cc52605f225,cf00c0855c5247c5] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:38:10.079 INFO 2458 --- [Check obproxy1,9db67cc52605f225,cf00c0855c5247c5] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:38:10.079 INFO 2458 --- [Check obproxy1,9db67cc52605f225,cf00c0855c5247c5] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:38:10.081 INFO 2458 --- [Collect obproxy1,55938d299aed835a,40a3b6636bb4fca6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:38:10.084 INFO 2458 --- [Collect obproxy1,55938d299aed835a,40a3b6636bb4fca6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:38:10.087 INFO 2458 --- [Collect obproxy1,55938d299aed835a,40a3b6636bb4fca6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:38:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:38:10.368 INFO 2458 --- [ocp-schedules-27,c914979c46215fbd,ac89a0bdade3ab6b] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:10.386 WARN 2458 --- [scanner-112221424-1,c914979c46215fbd,74c31e9018f34cba] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:10.631 INFO 2458 --- [alarm-detect-13,c914979c46215fbd,bf5199163b8b766e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536265 ~ 1761536280 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536280, value=36.784)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536280, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536280=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:10.687 INFO 2458 --- [ocp-async-4,7840e4690dfb11dc,38382db05424374f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:10.687 WARN 2458 --- [ocp-async-9,1e537b80b947e212,160bf720190c2a47] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:11.320 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505683) Duplicate entry '835370602992181248-1761536160' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505683) Duplicate entry '835370602992181248-1761536160' for key 'PRIMARY' 2025-10-27 11:38:11.400 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=1000, affectRows=1000 2025-10-27 11:38:11.401 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=1000, affectRows=1000 2025-10-27 11:38:11.405 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=994, validCount=992 2025-10-27 11:38:11.405 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=999, validCount=995 2025-10-27 11:38:11.426 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=1000, affectRows=1000 2025-10-27 11:38:11.430 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=996, validCount=992 2025-10-27 11:38:11.440 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=523, affectRows=523 2025-10-27 11:38:11.442 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=520, validCount=520 2025-10-27 11:38:11.451 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=558, affectRows=558 2025-10-27 11:38:11.453 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=556, validCount=552 2025-10-27 11:38:11.707 INFO 2458 --- [prometheus-minute-schedule-1,f9751b7e6da45849,c88a393acd574f4b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:11.714 INFO 2458 --- [prometheus-minute-schedule-1,f9751b7e6da45849,c88a393acd574f4b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:11.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:11.715578+08:00, lastProcessTime=2025-10-27T11:38:01.715417+08:00 2025-10-27 11:38:11.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:11.715578+08:00 2025-10-27 11:38:12.411 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536220, nodeCount=817, affectRows=817 2025-10-27 11:38:12.687 INFO 2458 --- [ocp-async-3,92aa5acbb6aa707a,7da09a0c79b6e0a5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:12.687 WARN 2458 --- [ocp-async-10,8422f26b49563179,f12e75e4221ae38d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:12.727 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:12.728 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:14.688 INFO 2458 --- [ocp-async-1,6ca83198c5719a73,a0777ce97f08cd3e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:14.688 WARN 2458 --- [ocp-async-7,f3a9ca3fa9079650,594a174a0df54897] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:14.707 INFO 2458 --- [prometheus-minute-schedule-1,d9268971c5f60230,943c461c12f5a37a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:14.713 INFO 2458 --- [prometheus-minute-schedule-1,d9268971c5f60230,943c461c12f5a37a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:15.729 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:15.730 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:15.990 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536220, nodeCount=1000, affectRows=1000 2025-10-27 11:38:15.994 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536220, nodeCount=1000, validCount=987 2025-10-27 11:38:16.089 INFO 2458 --- [ocp-schedules-11,982d3f4fd6757d7c,1d04a898c44c4179] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:16.093 INFO 2458 --- [ocp-schedules-11,982d3f4fd6757d7c,1d04a898c44c4179] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:16.559 INFO 2458 --- [ocp-schedules-7,6fc9f67f6ec4fef5,86524a0f40e151f2] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:38:16.559+08:00, clusters=[1, 2] 2025-10-27 11:38:16.562 INFO 2458 --- [ocp-schedules-7,6fc9f67f6ec4fef5,86524a0f40e151f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:38:16.565 INFO 2458 --- [ocp-schedules-7,6fc9f67f6ec4fef5,86524a0f40e151f2] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:38:16.688 WARN 2458 --- [ocp-async-9,05dd193c3dbb826d,33b292f87a72f881] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:16.688 INFO 2458 --- [ocp-async-6,a2924aa0231d278f,259a45879af48340] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:17.257 INFO 2458 --- [ocp-schedules-9,276348c931762280,59867fb87dfb3fce] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:17.347 INFO 2458 --- [ocp-schedules-28,b93b89a1ec43c61c,38a7e18d0b045bf9] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:38:17.355 INFO 2458 --- [ocp-schedules-28,b93b89a1ec43c61c,38a7e18d0b045bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:17.450 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:38:17.451 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.454 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:38:17.462 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.463 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.489 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.490 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.504 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.504 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.505 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:38:17.505 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.505 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.505 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.508 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:38:17.513 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.514 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.516 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.516 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.524 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.524 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.524 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.525 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.541 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.542 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.552 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.552 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.552 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.553 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.568 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.568 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.580 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.580 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.580 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.581 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:38:17.583 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.583 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.583 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.584 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.599 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.600 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.600 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.601 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.602 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.603 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.604 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.604 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.605 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.606 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.607 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.607 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.608 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:38:17.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.611 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.611 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:38:17.611 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.611 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.641 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.641 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.642 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.642 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:38:17.645 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.645 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.645 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.646 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:38:17.679 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.679 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.679 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.679 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:38:17.690 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:17.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:38:17.701 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.701 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:38:17.701 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.701 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:38:17.704 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:38:17.704 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:38:17.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.707 INFO 2458 --- [prometheus-minute-schedule-1,2170214936fe4573,be30db6286eaa6ce] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:17.707 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.708 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.708 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:38:17.710 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.710 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.711 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.711 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:38:17.713 INFO 2458 --- [prometheus-minute-schedule-1,2170214936fe4573,be30db6286eaa6ce] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:17.743 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.743 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.743 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.744 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:38:17.770 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.770 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.770 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.771 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:38:17.814 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.814 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.815 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.815 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.816 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.816 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.816 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.817 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.817 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.817 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.817 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.818 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.819 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.819 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.819 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.820 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.821 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.821 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.822 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.823 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.823 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.823 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.824 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.825 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.825 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.826 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.826 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.826 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.826 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.827 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.827 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.828 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.828 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.829 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.829 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.829 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.829 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.830 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:38:17.830 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.830 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.831 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.831 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:38:17.832 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.832 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.832 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.833 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:38:17.833 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:38:17.836 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:38:17.920 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:17.920 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.920 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:17.921 INFO 2458 --- [Sync all tenant information1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:17.921 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:17.921 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:38:17.946 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:38:17.950 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:38:17.950 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:38:17.950 INFO 2458 --- [ocp-schedules-28,b93b89a1ec43c61c,38a7e18d0b045bf9] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:38:18.545 INFO 2458 --- [ocp-async-8,1a5aa28a06ea282c,641326d05f536da2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:18.688 INFO 2458 --- [ocp-async-10,19100fc81daa4fc0,f3af0362f38124da] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:18.688 WARN 2458 --- [ocp-async-2,b6f799d9c5c272aa,bdc40533db5fe170] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:18.700 INFO 2458 --- [ocp-schedules-30,cf35324b2194228a,9d1076ebca09c577] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:38:18.700 INFO 2458 --- [ocp-schedules-30,cf35324b2194228a,9d1076ebca09c577] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:38:18.730 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:18.731 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:18.774 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:38:18.774 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:38:18.775 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:38:18.775 INFO 2458 --- [Collect all tenants major compactions6,cf35324b2194228a,c37e388bc899b18d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:38:18.775 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:38:18.775 INFO 2458 --- [Collect all tenants major compactions5,cf35324b2194228a,1e2d96cfbb1f8c9d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:38:18.811 INFO 2458 --- [Collect all tenants major compactions6,cf35324b2194228a,c37e388bc899b18d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.811 INFO 2458 --- [Collect all tenants major compactions5,cf35324b2194228a,1e2d96cfbb1f8c9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.812 INFO 2458 --- [Collect all tenants major compactions6,cf35324b2194228a,c37e388bc899b18d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:38:18.812 INFO 2458 --- [Collect all tenants major compactions5,cf35324b2194228a,1e2d96cfbb1f8c9d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:38:18.816 INFO 2458 --- [Collect all tenants major compactions5,cf35324b2194228a,1e2d96cfbb1f8c9d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:38:18.817 INFO 2458 --- [Collect all tenants major compactions6,cf35324b2194228a,c37e388bc899b18d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:38:18.827 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.827 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.828 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.830 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:38:18.830 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:38:18.830 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:38:18.833 INFO 2458 --- [Collect all tenants major compactions1,cf35324b2194228a,64a32c0ed4a8d60f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:38:18.833 INFO 2458 --- [Collect all tenants major compactions4,cf35324b2194228a,22a7656885f7b706] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:38:18.833 INFO 2458 --- [Collect all tenants major compactions3,cf35324b2194228a,0dc56d8991654b09] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:38:18.838 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.838 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.839 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:18.839 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:18.839 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:18.840 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:38:18.846 INFO 2458 --- [Collect all tenants major compactions2,cf35324b2194228a,56e67b1c304c2733] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:38:18.848 INFO 2458 --- [ocp-schedules-30,cf35324b2194228a,9d1076ebca09c577] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:38:19.087 INFO 2458 --- [ocp-schedules-28,3209e45e25a172d5,b8c7a8feeaa02d6c] c.o.o.s.common.scheduled.HostSchedules : Check host version schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:19.087 INFO 2458 --- [ocp-schedules-28,3209e45e25a172d5,b8c7a8feeaa02d6c] o.o.c.t.e.u.DistributedExclusiveExecutor : Empty item list, taskName=host-check-agent-version, skip execute 2025-10-27 11:38:20.678 INFO 2458 --- [ocp-schedules-23,a715f67d0a006f2a,4a3dbf2e95de5405] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:20.688 INFO 2458 --- [ocp-async-10,eb235a2b3d360a9d,fcb021f6528c4be2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:20.688 WARN 2458 --- [ocp-async-7,4fa21775606def8d,8b18b2e92e476f7d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:20.707 INFO 2458 --- [prometheus-minute-schedule-1,5091f16c38736959,1610d2e473f3c3c0] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:20.725 WARN 2458 --- [scanner-112223246-1,a715f67d0a006f2a,31f211d890183438] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:20.729 INFO 2458 --- [prometheus-minute-schedule-1,5091f16c38736959,1610d2e473f3c3c0] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:20.784 INFO 2458 --- [alarm-detect-15,a715f67d0a006f2a,d732ddcb39850c96] c.o.o.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, activeAt=2025-10-27T11:38:00.278+08:00 2025-10-27 11:38:20.942 INFO 2458 --- [ocp-schedules-4,e50d7a1c2e2fe459,1d613f0ebac6b6c1] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:38:21.410 INFO 2458 --- [ocp-schedules-27,a76d8b12f49f168a,9f5d3cfed6061349] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:21.411 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:38:21.411 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:38:21.411 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:38:21.411 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:38:21.412 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:38:21.425 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:21.427 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:21.427 INFO 2458 --- [ocp-schedules-20,01b4c4fb562d19e8,60313a766dc10b88] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:21.428 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:21.428 INFO 2458 --- [host-check-clockdiff2,01b4c4fb562d19e8,ef1d27488e0192ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:38:21.428 INFO 2458 --- [host-check-clockdiff3,01b4c4fb562d19e8,02609c536472e19a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:38:21.428 INFO 2458 --- [host-check-clockdiff4,01b4c4fb562d19e8,fd32c34342809757] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:38:21.428 INFO 2458 --- [host-check-clockdiff5,01b4c4fb562d19e8,f7382480034ed4f3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:38:21.429 INFO 2458 --- [host-check-clockdiff1,01b4c4fb562d19e8,4d849f81c65a0ce7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:38:21.429 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:21.431 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:21.439 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:21.440 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:21.447 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:21.447 INFO 2458 --- [ocp-schedules-11,6761a079ca5b4260,edef33792e6fdfa5] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:38:21.447 INFO 2458 --- [host-check-clockdiff3,01b4c4fb562d19e8,02609c536472e19a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536301 -1 -1 , error= 2025-10-27 11:38:21.449 INFO 2458 --- [host-check-clockdiff1,01b4c4fb562d19e8,4d849f81c65a0ce7] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536301 0 0 , error= 2025-10-27 11:38:21.450 INFO 2458 --- [host-check-clockdiff5,01b4c4fb562d19e8,f7382480034ed4f3] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536301 0 0 , error= 2025-10-27 11:38:21.450 INFO 2458 --- [host-check-clockdiff4,01b4c4fb562d19e8,fd32c34342809757] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536301 0 0 , error= 2025-10-27 11:38:21.457 INFO 2458 --- [ocp-schedules-16,6fec1adba6359b21,2fea0b08c1a6597b] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:21.457 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:21.457 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:38:21.458 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:38:21.458 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:38:21.458 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:38:21.459 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:38:21.462 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:21.475 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:21.475 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:21.476 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:21.477 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:21.478 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:21.479 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:21.479 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:21.481 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:21.482 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:21.484 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:21.546 INFO 2458 --- [ocp-schedules-26,4910a53838b0b0b1,3070a55844dee482] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:38:21.550 INFO 2458 --- [ocp-schedules-30,fc3ed6f435e8cb46,04489fe05f3753a0] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:38:21.553 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:38:21.553 INFO 2458 --- [ocp-schedules-26,4910a53838b0b0b1,3070a55844dee482] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:21.554 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:38:21.556 INFO 2458 --- [ocp-schedules-15,584ed4fb2510e013,2bf4270857f45d49] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:38:21.557 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:38:21.557 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:38:21.557 INFO 2458 --- [host-check-agent-status5,a76d8b12f49f168a,670ddbc0e9a878c1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:38:21.558 INFO 2458 --- [ocp-schedules-30,fc3ed6f435e8cb46,04489fe05f3753a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:21.560 INFO 2458 --- [ocp-schedules-15,584ed4fb2510e013,2bf4270857f45d49] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:21.561 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:38:21.562 INFO 2458 --- [ocp-schedules-30,fc3ed6f435e8cb46,04489fe05f3753a0] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status1,a76d8b12f49f168a,a1cc6f997c5c737f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:38:21.565 INFO 2458 --- [ocp-schedules-29,afbed312164d9b65,228e9aa4f45b2f00] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:38:21.565 INFO 2458 --- [host-check-agent-status3,a76d8b12f49f168a,b59fea11c9e6e9b6] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:38:21.566 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:38:21.566 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:38:21.566234291+08:00 2025-10-27 11:38:21.567 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:38:21.570 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:38:21.567+08:00 2025-10-27 11:38:21.570 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:38:21.570 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:38:21.570 INFO 2458 --- [host-check-agent-status4,a76d8b12f49f168a,9aa259f890f90866] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:38:21.571 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:38:21.572 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:38:21.572 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:38:21.571980170+08:00 2025-10-27 11:38:21.572 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:38:21.572 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:38:21.572965480+08:00 2025-10-27 11:38:21.573 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:38:21.572+08:00 2025-10-27 11:38:21.573 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:38:21.572+08:00 2025-10-27 11:38:21.573 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:38:21.572771313+08:00 2025-10-27 11:38:21.574 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:38:21.574+08:00 2025-10-27 11:38:21.579 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:38:21.567+08:00 and status AVAILABLE 2025-10-27 11:38:21.583 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:38:21.574+08:00 and status AVAILABLE 2025-10-27 11:38:21.583 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:38:21.572+08:00 and status AVAILABLE 2025-10-27 11:38:21.584 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:38:21.572+08:00 and status AVAILABLE 2025-10-27 11:38:21.587 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:38:21.590 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:38:21.590 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:38:21.590 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:38:21.591 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:38:21.590395024+08:00 2025-10-27 11:38:21.591 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:38:21.592 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:38:21.591+08:00 2025-10-27 11:38:21.601 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.602 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:21.603 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:38:21.604 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:38:21.591+08:00 and status AVAILABLE 2025-10-27 11:38:21.610 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:38:21.610 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:38:21.613 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:38:21.613 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:38:21.613 INFO 2458 --- [host-check-agent-status2,a76d8b12f49f168a,67b737923ed8166a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:38:21.621 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:21.622 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:21.622 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:21.625 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:21.626 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:21.626 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.628 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.629 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.629 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:21.634 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:38:21.634 INFO 2458 --- [ocp-schedules-26,4910a53838b0b0b1,3070a55844dee482] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:38:21.635 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:21.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.638 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:21.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:21.639 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:21.642 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.678 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.679 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:21.690 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:21.697 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:21.703 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:21.707 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:21.721 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:21.726 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:38:21.726 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:38:21.729 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:38:21.731 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:21.732 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:21.738 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:38:21.739 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:21.740 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.740 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:38:21.779 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:38:21.781 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:38:21.782 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:21.781973+08:00, lastProcessTime=2025-10-27T11:38:11.715578+08:00 2025-10-27 11:38:21.783 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:38:21.788 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:38:21.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:21.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.817 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.818 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.818 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:38:21.819 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:21.820 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:21.820 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:21.821 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:21.834 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:38:21.839 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:38:21.851 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:38:21.854 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:38:21.855 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:21.856 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.857 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:21.858 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.861 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:38:21.865 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:21.871 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:38:21.877 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:38:21.899 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:21.899 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:21.901 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:38:21.903 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:38:21.917 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:21.920 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:21.929 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:21.930 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:21.932 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:21.935 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:38:21.936 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:38:21.937 INFO 2458 --- [ocp-schedules-18,c0c0fdf590f17ee7,f5c7c1dbe36b6960] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:38:21.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:21.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.938 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.939 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:21.940 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:21.940 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:21.940 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.943 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:38:21.947 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:21.951 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:38:21.959 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:21.960 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:21.961 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:21.961 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:38:21.962 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:38:21.974 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:38:21.978 INFO 2458 --- [host-check-host-status5,6fec1adba6359b21,d301b5ef914c798e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:38:21.983 INFO 2458 --- [host-check-host-status4,6fec1adba6359b21,80eb00f1dfe48a06] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:38:21.984 INFO 2458 --- [host-check-host-status3,6fec1adba6359b21,813651ad1b7c9828] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:38:21.985 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:21.988 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:21.992 INFO 2458 --- [host-check-host-status1,6fec1adba6359b21,1cbbf794ac28dee4] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:38:22.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.004 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:22.048 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.049 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:22.063 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:38:22.065 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:21.781973+08:00 2025-10-27 11:38:22.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:22.107 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:38:22.107 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.108 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:38:22.125 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.125 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:22.135 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.136 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:38:22.144 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:38:22.153 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:38:22.161 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.161 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:22.171 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:38:22.176 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:22.176 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.177 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:38:22.206 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:22.206 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:22.207 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:38:22.208 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:38:22.227 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:38:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.244 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.245 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:22.252 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:38:22.253 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:38:22.261 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:38:22.270 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:22.273 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:22.277 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:22.301 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:38:22.328 INFO 2458 --- [host-check-host-status2,6fec1adba6359b21,4a3b3f519dc90a24] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:38:22.340 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:38:22.390 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:38:22.390 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.390 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:38:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.394 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:38:22.429 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:38:22.467 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:22.467 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.467 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.468 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:22.468 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:22.469 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:22.469 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:38:22.525 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:38:22.525 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:38:22.525 INFO 2458 --- [ocp-schedules-15,584ed4fb2510e013,2bf4270857f45d49] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:38:22.688 INFO 2458 --- [ocp-async-10,959123ab0c20ea19,fb450b80a68e7cd6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:22.688 WARN 2458 --- [ocp-async-9,c2dc9832cb509a72,ba64d80bbeae1eca] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:23.219 INFO 2458 --- [host-check-clockdiff2,01b4c4fb562d19e8,ef1d27488e0192ff] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536303 -1 -1 , error= 2025-10-27 11:38:23.707 INFO 2458 --- [prometheus-minute-schedule-1,369b23c24082f9d8,4f87e00e5370fc08] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:23.715 INFO 2458 --- [prometheus-minute-schedule-1,369b23c24082f9d8,4f87e00e5370fc08] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:24.688 WARN 2458 --- [ocp-async-10,4e27e379aa2880a8,dd84609062b0b04f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:24.688 INFO 2458 --- [ocp-async-10,a9feb30aba838863,f13ebd109a0d44cf] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:24.732 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:24.733 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:25.370 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36} 2025-10-27 11:38:25.382 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo= 2025-10-27 11:38:25.382 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:25.384 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:25.384 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:38:25.384 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:25.387 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:25.388 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:25.388 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:38:25.388 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:25.391 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:25.392 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:25.392 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:38:25.392 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:25.395 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:25.419 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:25.419 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:38:25.419 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:25.557 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449905557000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:38:25.557Z, endTime=2025-10-27T03:36:25.360Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:38:25.558 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761449905558000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:38:25.558Z, endTime=2025-10-27T03:36:25.360Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:38:25.592 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 264 2025-10-27 11:38:25.592 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:38:25.643 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:38:25.644 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:38:25.657 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:38:25.657 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:38:25.693 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:38:25.693 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:38:25.699 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:38:25.718 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:38:25.557, end=2025-10-27T11:36:25.36}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:38:26.055 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:38:26.055 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:38:26.088 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:38:26.089 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:38:26.098 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:38:26.098 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:38:26.114 INFO 2458 --- [ocp-schedules-16,74c1ef1b72ee4a3e,57d9c2aaac2e65c4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:26.117 INFO 2458 --- [ocp-schedules-16,74c1ef1b72ee4a3e,57d9c2aaac2e65c4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:26.134 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:38:26.134 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536185360000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:38:26.139 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:38:26.140 INFO 2458 --- [plan_change_detector5,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:38:25.558, end=2025-10-27T11:36:25.36}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:38:26.140 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:26.141 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:26.141 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:38:26.141 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:26.144 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:38:26.147 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:34:24.672, end=2025-10-27T11:36:25.36},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:38:26.147 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:38:26.147 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:38:26.153 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:38:26.688 INFO 2458 --- [ocp-async-10,9ff0a9a728f9bee4,fb5c268384f1382b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:26.688 WARN 2458 --- [ocp-async-5,4e2a8d48a7675d27,6785921545202705] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:26.707 INFO 2458 --- [prometheus-minute-schedule-1,0fdea66282399526,e3183daf7086c4d6] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:26.743 INFO 2458 --- [prometheus-minute-schedule-1,0fdea66282399526,e3183daf7086c4d6] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:27.263 INFO 2458 --- [ocp-schedules-15,75f710c6bc4a3dfe,b4c64497cd29dbe2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:27.733 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:27.734 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:27.916 INFO 2458 --- [ocp-schedules-4,01e9614198bd4e42,f9300e560130f1ed] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:38:27.935 INFO 2458 --- [ocp-schedules-4,01e9614198bd4e42,f9300e560130f1ed] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:27.935 INFO 2458 --- [ocp-schedules-4,01e9614198bd4e42,f9300e560130f1ed] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:28.545 INFO 2458 --- [ocp-async-1,f16244edf5b9a715,708049d013d5d4c1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:28.689 WARN 2458 --- [ocp-async-9,5ce8a5cd40d32a00,131d0bcc67521b12] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:28.689 INFO 2458 --- [ocp-async-6,a6296ca63549fceb,48cdf63dddba0c7e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:29.104 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:38:29.104 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:38:29.105 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:38:29.105 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839265989532524544 2025-10-27 11:38:29.107 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:38:29.107 INFO 2458 --- [ocp-schedules-30,cfa03dd6e0a7c06d,7e4265ca12094499] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:38:29.549 INFO 2458 --- [ocp-schedules-17,30761875d42afe8c,9b4bc01ea98d660c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:38:29.564 INFO 2458 --- [ocp-schedules-17,30761875d42afe8c,9b4bc01ea98d660c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:38:29.564 INFO 2458 --- [ocp-schedules-17,30761875d42afe8c,9b4bc01ea98d660c] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:38:29.707 INFO 2458 --- [prometheus-minute-schedule-1,79b361f9806d8971,32d044e21907cdcb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:29.716 INFO 2458 --- [prometheus-minute-schedule-1,79b361f9806d8971,32d044e21907cdcb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:30.689 WARN 2458 --- [ocp-async-6,ba95403472a4c6d7,2f7b53356c321926] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:30.689 INFO 2458 --- [ocp-async-6,215bc83da3d6acd6,bb02ffdc4a6817ca] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:30.734 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:30.735 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:30.956 INFO 2458 --- [ocp-schedules-4,8d96fa36675441e0,a1c5448bcd344a8f] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:31.054 WARN 2458 --- [scanner-112223390-1,8d96fa36675441e0,aaf4e5e39fcb4d33] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:31.309 INFO 2458 --- [alarm-detect-2,8d96fa36675441e0,89a5f9ee7e4bd139] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536280 ~ 1761536301 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536295, value=1.55)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.55, values={1761536295=1.55}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:31.310 INFO 2458 --- [alarm-detect-2,8d96fa36675441e0,89a5f9ee7e4bd139] c.o.o.a.c.detect.AlarmRuleDetectState : firstDetect, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, currentState=Pending 2025-10-27 11:38:31.608 INFO 2458 --- [alarm-detect-12,8d96fa36675441e0,a37a8b864ea82414] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536280 ~ 1761536301 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536295, value=49.792)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536295, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536295=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:31.736 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:31.735065+08:00, lastProcessTime=2025-10-27T11:38:21.781973+08:00 2025-10-27 11:38:32.309 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:31.735065+08:00 2025-10-27 11:38:32.689 WARN 2458 --- [ocp-async-7,fa1a138132c99917,38aba336a16b0995] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:32.689 INFO 2458 --- [ocp-async-1,6f147564b64f61be,25e620b20e427757] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:32.707 INFO 2458 --- [prometheus-minute-schedule-1,c33be15a920c2461,73253b055a3705c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:32.719 INFO 2458 --- [prometheus-minute-schedule-1,c33be15a920c2461,73253b055a3705c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:33.735 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:33.736 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:34.021 INFO 2458 --- [ocp-schedules-22,007f9ca735712786,a123167cf203e925] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:38:34.021 INFO 2458 --- [ocp-schedules-22,007f9ca735712786,a123167cf203e925] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:38:34.689 INFO 2458 --- [ocp-async-10,0ee63f07045d3458,ad6527bfe5d32f51] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:34.689 WARN 2458 --- [ocp-async-8,9d399e834b52b707,560456ac5a26f818] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:35.707 INFO 2458 --- [prometheus-minute-schedule-1,d5b2f4bb9ceb2a43,a3e91d797dfcd068] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:35.717 INFO 2458 --- [prometheus-minute-schedule-1,d5b2f4bb9ceb2a43,a3e91d797dfcd068] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:36.125 INFO 2458 --- [ocp-schedules-29,00e37c423fadf61f,3606bcf11f3db850] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:36.131 INFO 2458 --- [ocp-schedules-29,00e37c423fadf61f,3606bcf11f3db850] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:36.689 WARN 2458 --- [ocp-async-4,e747d964e2eed6b4,8e26dba1936bbda3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:36.689 INFO 2458 --- [ocp-async-5,89f0401c8db80815,38af1a95d4718266] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:36.736 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:36.738 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:37.283 INFO 2458 --- [ocp-schedules-27,45c847d22dab27d6,a8d9c6eff0530330] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:37.708 INFO 2458 --- [prometheus-cache-monitor1,b6d963387b80f2a4,fb90c646bcb2f672] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20322,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5080,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:38:37.708 INFO 2458 --- [prometheus-cache-monitor1,b6d963387b80f2a4,fb90c646bcb2f672] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:38:38.545 INFO 2458 --- [ocp-async-5,0b5991de601b573c,c69699a2f23f83fe] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:38.689 INFO 2458 --- [ocp-async-6,27b71b4290e99a12,d348c0a398c7ed34] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:38.689 WARN 2458 --- [ocp-async-1,b7ff360f155a0810,aed0807c930197ad] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:38.707 INFO 2458 --- [prometheus-minute-schedule-1,f1e166e1cbb66472,9678f97420bb9b7f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:38.715 INFO 2458 --- [prometheus-minute-schedule-1,f1e166e1cbb66472,9678f97420bb9b7f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:38.943 INFO 2458 --- [ocp-schedules-23,b99d0f72fc06b37e,5245d5e60ce0d52a] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:38:38.951 INFO 2458 --- [ocp-schedules-23,b99d0f72fc06b37e,5245d5e60ce0d52a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:38.951 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:38:38.951 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:38:38.952 INFO 2458 --- [ocp-schedules-23,b99d0f72fc06b37e,5245d5e60ce0d52a] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:38:39.738 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:39.739 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:40.043 INFO 2458 --- [ocp-schedules-8,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:38:40.043 INFO 2458 --- [ocp-schedules-17,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:38:40.120 INFO 2458 --- [Check obproxy1,955a71210dbaef4b,20bb0889b41871af] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:38:40.120 INFO 2458 --- [Check obproxy1,955a71210dbaef4b,20bb0889b41871af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:38:40.120 INFO 2458 --- [Collect obproxy1,a85399d005e918a1,670dd0a53b4d5810] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:38:40.121 INFO 2458 --- [Collect obproxy1,a85399d005e918a1,670dd0a53b4d5810] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:38:40.151 INFO 2458 --- [Collect obproxy1,a85399d005e918a1,670dd0a53b4d5810] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:38:40.154 INFO 2458 --- [Check obproxy1,955a71210dbaef4b,20bb0889b41871af] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:38:40.154 INFO 2458 --- [Collect obproxy1,a85399d005e918a1,670dd0a53b4d5810] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:38:40.157 INFO 2458 --- [Check obproxy1,955a71210dbaef4b,20bb0889b41871af] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:38:40.157 INFO 2458 --- [Check obproxy1,955a71210dbaef4b,20bb0889b41871af] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:38:40.157 INFO 2458 --- [Collect obproxy1,a85399d005e918a1,670dd0a53b4d5810] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:38:40.513 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:40.513 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:38:40.513 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:38:40.513 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:38:40.513 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:38:40.515 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:38:40.525 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:40.525 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:40.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:40.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:40.526 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:40.532 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:40.533 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:40.534 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:40.534 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:40.534 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:40.660 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:38:40.664 INFO 2458 --- [ForkJoinPool.commonPool-worker-0,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:38:40.681 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:38:40.689 INFO 2458 --- [ForkJoinPool.commonPool-worker-6,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:38:40.690 INFO 2458 --- [ocp-async-9,20cdf8a9eaed3eef,f7c0adff33605b1c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:40.690 WARN 2458 --- [ocp-async-2,7ac57b28fe43621b,aca3616acadec5bf] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:40.706 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:38:40.708 INFO 2458 --- [ocp-schedules-12,fc8bdfd71d8000f6,cce78c8bba0520f4] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:38:41.128 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:38:41.128 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:38:41.234 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:41.234 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:41.283 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:41.283 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:41.283 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:41.284 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:38:41.296 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:41.296 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:41.296 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:41.297 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:38:41.305 INFO 2458 --- [ocp-schedules-10,02a29a37aad8a61b,3a731e80a2b50420] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:38:41.364 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:41.365 INFO 2458 --- [ocp-schedules-3,61f03d6aeff59dc3,5d9c0378f894aef9] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:38:41.621 INFO 2458 --- [ocp-schedules-17,d0ea69ad8e48a948,d0a7aad9ed810555] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:41.644 WARN 2458 --- [scanner-112223666-1,d0ea69ad8e48a948,7dcc29d9c805fbee] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:41.660 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.s.common.scheduled.OcpSchedules : Refresh OCP properties. 2025-10-27 11:38:41.660 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.ocp.core.property.PropertyService : Refresh config properties 2025-10-27 11:38:41.665 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:38:41.665 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:38:41.665 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:38:41.665 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:38:41.669 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : new: [] 2025-10-27 11:38:41.669 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : init spring 2025-10-27 11:38:41.669 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : have already run initialize. 2025-10-27 11:38:41.669 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : starting 2025-10-27 11:38:41.707 INFO 2458 --- [prometheus-minute-schedule-1,d0e9ea694d91e2a4,7f0d05a8b519bf1a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:41.714 INFO 2458 --- [prometheus-minute-schedule-1,d0e9ea694d91e2a4,7f0d05a8b519bf1a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:41.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:41.718949+08:00, lastProcessTime=2025-10-27T11:38:31.735065+08:00 2025-10-27 11:38:41.730 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:38:41.736 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:38:41.736 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:38:41.736 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:38:41.736 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:38:41.736 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:38:41.742 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:38:42.121 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.DataSourceInterceptor : dataSource dataSource already initialized. skipped. class: org.springframework.jdbc.datasource.DriverManagerDataSource 2025-10-27 11:38:42.176 INFO 2458 --- [alarm-detect-19,d0ea69ad8e48a948,3713924696869430] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536295 ~ 1761536311 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536310, value=55.08)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536310, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536310=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:42.356 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:41.718949+08:00 2025-10-27 11:38:42.521 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT0.85128952S 2025-10-27 11:38:42.521 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : environmentPrepared 2025-10-27 11:38:42.577 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [config/ocp.yaml]' via location 'classpath:/config/' 2025-10-27 11:38:42.577 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] o.s.c.c.s.e.NativeEnvironmentRepository : Adding property source: Config resource 'class path resource [application.properties]' via location 'classpath:/' 2025-10-27 11:38:42.626 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] b.c.PropertySourceBootstrapConfiguration : Located property source: [BootstrapPropertySource {name='bootstrapProperties-classpath:/config/ocp.yaml'}, BootstrapPropertySource {name='bootstrapProperties-classpath:/application.properties'}, BootstrapPropertySource {name='bootstrapProperties-ocp-default'}] 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : contextPrepared 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : set internal properties as the first PropertySource 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : register bean ProgressHandler 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : set internal property ocp.version to 4.3.6 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : ocp.version in spring: 4.3.6 2025-10-27 11:38:42.675 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] o.s.boot.SpringApplication : The following 1 profile is active: "composite" 2025-10-27 11:38:42.685 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : contextLoaded 2025-10-27 11:38:42.690 INFO 2458 --- [ocp-async-10,94ced5141f6692ae,e240f362628852f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:42.690 WARN 2458 --- [ocp-async-5,bf8d1eeaec56c189,baf4deba91425ef0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:42.739 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:42.740 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:42.815 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] o.s.boot.SpringApplication : Started application in 1.149 seconds (JVM running for 258791.242) 2025-10-27 11:38:42.815 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.o.b.spring.BootstrapRunListener : ready. time taken: PT1.149712515S 2025-10-27 11:38:43.020 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,58c199950520b381,18f6ca3460d97326] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=58c199950520b381, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:38:43.021 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,58c199950520b381,18f6ca3460d97326] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=58c199950520b381, duration=1 ms] 2025-10-27 11:38:43.207 INFO 2458 --- [ocp-schedules-15,28f34c88999ef5e3,78706a7500c7fe4f] c.o.ocp.core.property.PropertyService : Refresh config properties done 2025-10-27 11:38:44.342 INFO 2458 --- [ocp-schedules-3,aab5d7b50f7d34b3,d90f6ae8a18d4148] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:38:44.690 WARN 2458 --- [ocp-async-1,bde1f234213e7767,f706e13169db5e40] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:44.690 INFO 2458 --- [ocp-async-7,f39f01eeb3d929c8,b6a912314699b5de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:44.707 INFO 2458 --- [prometheus-minute-schedule-1,6c7904967d82fb2f,d9fef166c1611f7b] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:44.717 INFO 2458 --- [prometheus-minute-schedule-1,6c7904967d82fb2f,d9fef166c1611f7b] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:45.102 INFO 2458 --- [prometheus-second-schedule-1,afe9cca548c38d2d,f767ec7599f957c4] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:38:45.740 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:45.741 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:46.142 INFO 2458 --- [ocp-schedules-16,08c71e4f76268a7c,ea550829d34ee1e6] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:46.146 INFO 2458 --- [ocp-schedules-16,08c71e4f76268a7c,ea550829d34ee1e6] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:46.690 INFO 2458 --- [ocp-async-2,b6bc339f42fd2dbd,708912b7a7d481b3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:46.690 WARN 2458 --- [ocp-async-6,5608f0f9df095e0a,114ae8c59e8178bb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:47.292 INFO 2458 --- [ocp-schedules-5,8403e8ee14182982,a03a183956a87c64] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:47.707 INFO 2458 --- [prometheus-minute-schedule-1,958f18cd28e01845,2b3c3ecbd77510a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:47.714 INFO 2458 --- [prometheus-minute-schedule-1,958f18cd28e01845,2b3c3ecbd77510a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:47.939 INFO 2458 --- [ocp-schedules-4,d74ea2d070f63c2c,33cd822c400060dc] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:38:47.949 INFO 2458 --- [ocp-schedules-4,d74ea2d070f63c2c,33cd822c400060dc] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:47.949 INFO 2458 --- [ocp-schedules-4,d74ea2d070f63c2c,33cd822c400060dc] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:38:48.545 INFO 2458 --- [ocp-async-2,ee606e619456e82e,8f2e93e1738c0cc3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:48.690 WARN 2458 --- [ocp-async-4,2e7101cbc19f60e2,95ca9ced3b6ac17f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:48.690 INFO 2458 --- [ocp-async-5,ffdd01e7b9b58005,fd1259fd79c44fb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:48.742 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:48.743 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:50.278 INFO 2458 --- [ocp-schedules-8,cc8927eef649d293,f6db8533301ea440] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:38:50.278+08:00, clusters=[1, 2] 2025-10-27 11:38:50.298 INFO 2458 --- [ocp-schedules-8,cc8927eef649d293,f6db8533301ea440] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:38:50.330 INFO 2458 --- [ocp-schedules-8,cc8927eef649d293,f6db8533301ea440] c.o.o.p.s.i.alarm.SlowSqlDataSource : query SlowSqlData of obCluster myoceanbase2, exclude result is empty. 2025-10-27 11:38:50.690 WARN 2458 --- [ocp-async-7,cea94fe59a95722d,2e8d469aadbfb342] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:50.690 INFO 2458 --- [ocp-async-1,e75ff9e93caf1622,3f0aad0a97a0acb9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:50.707 INFO 2458 --- [prometheus-minute-schedule-1,5562bfdee4166690,896552cf9ed9b74d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:50.713 INFO 2458 --- [prometheus-minute-schedule-1,5562bfdee4166690,896552cf9ed9b74d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:51.409 INFO 2458 --- [ocp-schedules-20,27de8b7f24c2cfa9,43db1831db43d465] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:51.409 INFO 2458 --- [ocp-schedules-27,0a7232703596588c,7f75bce45d005c41] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:38:51.409 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:38:51.410 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:38:51.410 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:38:51.410 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:38:51.425 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:51.425 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:51.426 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:51.426 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:51.427 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:51.427 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:51.427 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:51.428 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:51.429 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:51.430 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:51.430 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:51.430 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:51.431 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:51.431 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:51.432 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:51.432 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:51.432 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:51.433 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:51.438 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:51.439 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:51.442 INFO 2458 --- [ocp-schedules-23,a5b2196ed0e0187c,e25c1c6b0668e764] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:38:51.442 INFO 2458 --- [ocp-schedules-7,b03e2b1078343dea,538bf816cda735a6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:38:51.443 INFO 2458 --- [host-check-clockdiff1,b03e2b1078343dea,c94f6dfba74cdbf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:38:51.444 INFO 2458 --- [host-check-clockdiff3,b03e2b1078343dea,e2d33d46749e8cec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:38:51.445 INFO 2458 --- [host-check-clockdiff2,b03e2b1078343dea,a2623a78d8424ab2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:38:51.447 INFO 2458 --- [host-check-clockdiff4,b03e2b1078343dea,00c1397fc3bccd22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:38:51.448 INFO 2458 --- [host-check-clockdiff5,b03e2b1078343dea,bc2ffb7bce1b2a91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:38:51.479 INFO 2458 --- [host-check-clockdiff3,b03e2b1078343dea,e2d33d46749e8cec] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536331 -1 -1 , error= 2025-10-27 11:38:51.520 INFO 2458 --- [host-check-clockdiff5,b03e2b1078343dea,bc2ffb7bce1b2a91] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536331 0 0 , error= 2025-10-27 11:38:51.520 INFO 2458 --- [host-check-clockdiff1,b03e2b1078343dea,c94f6dfba74cdbf4] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536331 0 0 , error= 2025-10-27 11:38:51.520 INFO 2458 --- [host-check-clockdiff2,b03e2b1078343dea,a2623a78d8424ab2] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536331 -1 -1 , error= 2025-10-27 11:38:51.532 INFO 2458 --- [host-check-clockdiff4,b03e2b1078343dea,00c1397fc3bccd22] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536331 0 0 , error= 2025-10-27 11:38:51.545 INFO 2458 --- [ocp-schedules-24,31d579938fc4aed7,77d454c50d0666e2] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:38:51.549 INFO 2458 --- [ocp-schedules-8,d2a4bd69c48602e1,9782b94dcae9e68a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:38:51.551 INFO 2458 --- [ocp-schedules-24,31d579938fc4aed7,77d454c50d0666e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:51.554 INFO 2458 --- [ocp-schedules-4,4cc91ea6c718bd61,ed3a173a07e7ad84] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:38:51.559 INFO 2458 --- [ocp-schedules-4,4cc91ea6c718bd61,ed3a173a07e7ad84] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:51.568 INFO 2458 --- [ocp-schedules-16,621f1c17f66e0bc6,3b2e5cd5af9742f0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:38:51.568 INFO 2458 --- [ocp-schedules-8,d2a4bd69c48602e1,9782b94dcae9e68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:38:51.570 INFO 2458 --- [ocp-schedules-1,911e53a79c3ba1af,38bc93aaf2aa4967] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:38:51.571 INFO 2458 --- [ocp-schedules-21,5cdb91d02fe35622,43d1fc46fcb7d041] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:38:51.575 INFO 2458 --- [ocp-schedules-8,d2a4bd69c48602e1,9782b94dcae9e68a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:38:51.576 INFO 2458 --- [ocp-schedules-16,621f1c17f66e0bc6,3b2e5cd5af9742f0] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:38:51.587 INFO 2458 --- [ocp-schedules-1,911e53a79c3ba1af,38bc93aaf2aa4967] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:38:51.625 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:38:51.626 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:38:51.626 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:38:51.628 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:38:51.628833933+08:00 2025-10-27 11:38:51.629 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:38:51.629 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:38:51.629 INFO 2458 --- [host-check-agent-status4,0a7232703596588c,4bd30fdafef62f2d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:38:51.629 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:38:51.628+08:00 2025-10-27 11:38:51.629 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:38:51.628202596+08:00 2025-10-27 11:38:51.630 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:38:51.629+08:00 2025-10-27 11:38:51.646 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.648 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:51.648 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:38:51.628+08:00 and status AVAILABLE 2025-10-27 11:38:51.649 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:38:51.651 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:38:51.629+08:00 and status AVAILABLE 2025-10-27 11:38:51.656 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:38:51.656 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:38:51.658 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:38:51.660 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:38:51.660 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:38:51.660 INFO 2458 --- [host-check-agent-status5,0a7232703596588c,320ca6686ac9faf2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:38:51.675 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:38:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.677 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:38:51.677393954+08:00 2025-10-27 11:38:51.677 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.678 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:51.678 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:38:51.677+08:00 2025-10-27 11:38:51.684 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:38:51.684 INFO 2458 --- [ocp-schedules-24,31d579938fc4aed7,77d454c50d0666e2] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:38:51.692 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:38:51.693 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.693 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:51.696 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:38:51.696 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:38:51.696 INFO 2458 --- [host-check-agent-status3,0a7232703596588c,a817a4fb07e6770d] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:38:51.700 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:38:51.677+08:00 and status AVAILABLE 2025-10-27 11:38:51.702 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:38:51.706 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:38:51.710 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:51.712 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:51.714 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:38:51.714 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:51.714 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:38:51.714294650+08:00 2025-10-27 11:38:51.715 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:38:51.715 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:38:51.715 INFO 2458 --- [host-check-agent-status2,0a7232703596588c,33c04be3ff2938fa] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:38:51.715 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:38:51.714+08:00 2025-10-27 11:38:51.719 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:38:51.719 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:38:51.719269056+08:00 2025-10-27 11:38:51.720 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:38:51.720+08:00 2025-10-27 11:38:51.722 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:38:51.720574+08:00, lastProcessTime=2025-10-27T11:38:41.718949+08:00 2025-10-27 11:38:51.728 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:38:51.714+08:00 and status AVAILABLE 2025-10-27 11:38:51.730 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:38:51.720+08:00 and status AVAILABLE 2025-10-27 11:38:51.736 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:51.737 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:38:51.738 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:38:51.741 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:38:51.741 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:38:51.743 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:51.743 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:51.744 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.744 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.745 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.745 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.745 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.746 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:51.755 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:38:51.759 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:38:51.759 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:38:51.759 INFO 2458 --- [host-check-agent-status1,0a7232703596588c,812add809caa11bf] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:38:51.759 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:51.764 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:38:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.764 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:38:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:38:51.767 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:38:51.767 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:38:51.798 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:51.803 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:51.804 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:51.816 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:38:51.821 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:51.823 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:38:51.823 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:38:51.828 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:51.829 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.830 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.831 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.833 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:38:51.834 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:38:51.834 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.836 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:38:51.837 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:38:51.838 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:51.845 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:38:51.857 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:38:51.867 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:51.867 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:51.871 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:38:51.872 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:38:51.874 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:38:51.891 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:38:51.901 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:51.912 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:51.913 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:51.914 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:51.914 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:51.922 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:38:51.929 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:51.929 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:38:51.929 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:38:51.929 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.930 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:51.931 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:38:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.931 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.932 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.933 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.934 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:38:51.934 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:38:51.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:51.935 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:51.935 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:51.938 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:38:51.939 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:51.941 INFO 2458 --- [ocp-schedules-2,88599f186abd344d,98f0b7a867f84d03] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:38:51.943 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:38:51.944 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:38:51.952 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:38:51.956 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:38:51.966 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:38:51.975 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:38:51.976 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:38:51.985 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:38:51.987 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:38:51.988 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:38:51.998 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:38:51.720574+08:00 2025-10-27 11:38:51.999 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:51.999 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:52.005 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:52.007 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:52.016 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:52.018 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:52.020 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:38:52.023 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:38:52.023 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:38:52.035 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:38:52.037 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:38:52.044 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:38:52.046 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:38:52.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:52.050 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:52.057 INFO 2458 --- [host-check-host-status4,27de8b7f24c2cfa9,253ab5c516cd0035] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:38:52.060 INFO 2458 --- [host-check-host-status5,27de8b7f24c2cfa9,69291863d3ee9b8e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:38:52.073 INFO 2458 --- [host-check-host-status2,27de8b7f24c2cfa9,36af6cf2fa3ebaa1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:38:52.077 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:38:52.078 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.078 INFO 2458 --- [host-check-host-status1,27de8b7f24c2cfa9,ce8a0ea845fdf507] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:38:52.079 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:38:52.084 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.085 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:38:52.085 INFO 2458 --- [host-check-host-status3,27de8b7f24c2cfa9,d284199ded08af7f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:38:52.097 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:52.097 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:52.107 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:38:52.130 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:52.135 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.136 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:38:52.140 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:52.141 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:38:52.161 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:38:52.161 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:38:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:52.177 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:38:52.183 INFO 2458 --- [ocp-schedules-10,b22f2d97f50a2f87,788ad5bec195f64d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:38:52.225 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:38:52.227 WARN 2458 --- [scanner-112223852-1,b22f2d97f50a2f87,953902b5c82db6e7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:38:52.283 INFO 2458 --- [alarm-detect-2,b22f2d97f50a2f87,f8a55b5cb941bd48] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536295 ~ 1761536322 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536310, value=1.827)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.827, values={1761536310=1.827}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:38:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.285 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.288 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.289 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.290 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:38:52.296 INFO 2458 --- [alarm-detect-14,b22f2d97f50a2f87,6ae47981684b6104] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536262 ~ 1761536322 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536280, value=204614.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204614.56, values={1761536280=204614.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:38:52.298 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:38:52.353 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 2025-10-27 11:38:52.357 INFO 2458 --- [alarm-detect-10,b22f2d97f50a2f87,86a3065d29118fcc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536262 ~ 1761536322 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536280, value=93.26)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.26, values={1761536280=93.26}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:38:52.360 INFO 2458 --- [alarm-detect-18,b22f2d97f50a2f87,05e4008ea62171e4] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536262 ~ 1761536322 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536280, value=204615.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204615.912, values={1761536280=204615.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:38:52.367 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:38:52.370 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:38:52.423 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:38:52.470 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 2, sqlPort: 2881 got ObProcessInfoResult(pid=27089, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:38:52.471 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.471 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.471 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.471 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:38:52.473 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.473 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.473 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.474 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:38:52.513 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:38:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:38:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:38:52.552 INFO 2458 --- [Sync all clusters1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=ocp_monitor, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:38:52.553 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:38:52.553 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:38:52.593 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:38:52.593 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:38:52.594 INFO 2458 --- [ocp-schedules-4,4cc91ea6c718bd61,ed3a173a07e7ad84] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:38:52.690 INFO 2458 --- [ocp-async-2,994e83087868e4a7,5927d3c75a0dd4f2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:52.691 WARN 2458 --- [ocp-async-9,7a329df2d9bbc864,ba016770e5ed0028] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:38:53.707 INFO 2458 --- [prometheus-minute-schedule-1,d8855800542f349d,ada4084a88e28e1f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:53.714 INFO 2458 --- [prometheus-minute-schedule-1,d8855800542f349d,ada4084a88e28e1f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:54.604 INFO 2458 --- [ocp-schedules-29,4ee2a9486b9b6a56,ce401966f9c8d4f2] c.o.o.a.service.OcpAlarmInspectService : delete inactive alarm event count 0 2025-10-27 11:38:54.604 INFO 2458 --- [ocp-schedules-29,4ee2a9486b9b6a56,ce401966f9c8d4f2] c.o.o.a.service.OcpAlarmInspectService : deleteInactiveAlarmEvents count 0 2025-10-27 11:38:54.691 WARN 2458 --- [ocp-async-5,4d1463662aef0453,bd149d7ca28d8d7c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:54.691 INFO 2458 --- [ocp-async-7,4e2e5e62fa39da0f,52952bc6e3210ec1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:54.743 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:54.744 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:55.898 INFO 2458 --- [ocp-schedules-25,8fcd0b7a861eb0a5,2ae65a240e239145] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:38:56.154 INFO 2458 --- [ocp-schedules-24,fcaa9c56e03152f2,2cf916dc7271a8fd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:38:56.158 INFO 2458 --- [ocp-schedules-24,fcaa9c56e03152f2,2cf916dc7271a8fd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:38:56.691 WARN 2458 --- [ocp-async-6,48bdfc82169c1082,2f5ba1ebfc7a2123] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:56.691 INFO 2458 --- [ocp-async-1,84d438827fdb974b,92d036da5f5c4373] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:56.707 INFO 2458 --- [prometheus-minute-schedule-1,9f2b7fef1439c202,9565f60cfa476c02] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:56.713 INFO 2458 --- [prometheus-minute-schedule-1,9f2b7fef1439c202,9565f60cfa476c02] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:56.863 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] c.o.o.s.c.c.s.utils.CsrfRequestMatcher : CSRF token required for API [POST /api/v2/software-packages, client 10.40.198.13, traceId ffb81588d0a4e3e1] 2025-10-27 11:38:57.298 INFO 2458 --- [ocp-schedules-9,60371ae408e91e60,dd54bbedd61c3ab0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:38:57.744 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:38:57.745 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:38:58.545 INFO 2458 --- [ocp-async-3,477fbc69ccf03ade,260e4dab0f83de6f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:38:58.691 WARN 2458 --- [ocp-async-4,9db35fa2d40b70bb,e583ebc320d6c6e9] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:38:58.691 INFO 2458 --- [ocp-async-2,703b3f8764ee1969,b8849007ad90e982] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:38:59.622 INFO 2458 --- [ocp-schedules-17,dcd527ffe480ee57,6edfdbf64e69c219] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:38:59.707 INFO 2458 --- [prometheus-minute-schedule-1,18b50db7d31dabbd,d99b48141d6e20d8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536340, interval=60 2025-10-27 11:38:59.714 INFO 2458 --- [prometheus-minute-schedule-1,18b50db7d31dabbd,d99b48141d6e20d8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:38:59.868 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,b7b5dc84dfc3e1c1,515ef90c80b77c89] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=b7b5dc84dfc3e1c1, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:38:59.868 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-5,b7b5dc84dfc3e1c1,515ef90c80b77c89] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=b7b5dc84dfc3e1c1, duration=1 ms] 2025-10-27 11:39:00.691 INFO 2458 --- [ocp-async-10,295d326b0d2f7570,4d6de5957ea65a97] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:00.691 WARN 2458 --- [ocp-async-7,db4bfa7ee3a3c343,572d60fe3fa8d5be] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:00.745 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:00.746 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:01.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:01.715358+08:00, lastProcessTime=2025-10-27T11:38:51.720574+08:00 2025-10-27 11:39:01.948 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:01.715358+08:00 2025-10-27 11:39:02.391 INFO 2458 --- [ocp-schedules-22,272b9b6f0a28ac93,ce08d3a2f39c29ce] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:02.452 WARN 2458 --- [scanner-112224064-1,272b9b6f0a28ac93,b812d2d32191d032] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:02.554 INFO 2458 --- [alarm-detect-1,272b9b6f0a28ac93,d92f55b8873797f8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536310 ~ 1761536332 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536325, value=1.875)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.875, values={1761536325=1.875}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:02.691 WARN 2458 --- [ocp-async-1,2bbd257685748ee5,f43989b197758a74] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:02.691 INFO 2458 --- [ocp-async-1,457b458087e0ac5e,66517adc883d0501] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:02.708 INFO 2458 --- [prometheus-minute-schedule-1,06ca0e6ddc301a82,e7a1d75f357a53a1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:02.720 INFO 2458 --- [prometheus-minute-schedule-1,06ca0e6ddc301a82,e7a1d75f357a53a1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:02.720 INFO 2458 --- [alarm-detect-8,272b9b6f0a28ac93,26d37ab34235b4b9] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536310 ~ 1761536332 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536325, value=64.077)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536325, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536325=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:02.723 INFO 2458 --- [alarm-detect-8,272b9b6f0a28ac93,26d37ab34235b4b9] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_request_queue_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor:host=10.40.65.146-3] 2025-10-27 11:39:03.746 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:03.748 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:04.691 WARN 2458 --- [ocp-async-2,4e1907ff8830284e,74247cd2a9f8410d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:04.691 INFO 2458 --- [ocp-async-9,562708508a063439,46f12e71857faf22] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:05.144 INFO 2458 --- [ocp-schedules-9,383552319d4a3888,3d9c3a6f3c167467] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:39:05.152 INFO 2458 --- [ocp-schedules-9,383552319d4a3888,3d9c3a6f3c167467] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:05.192 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:05.193 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:39:05.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:39:05.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:39:05.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.65.168, port=2881, database=oceanbase) 2025-10-27 11:39:05.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.o.obsdk.connector.ObConnectorHolder : [obsdk] no ob connector found in holder, key=ObConnectorKey(connectionMode=direct, clusterName=myoceanbase, obClusterId=1760510187, tenantName=sys, username=root, address=10.40.67.22, port=2881, database=oceanbase) 2025-10-27 11:39:05.207 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:05.208 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:39:05.223 INFO 2458 --- [ocp-schedules-9,383552319d4a3888,3d9c3a6f3c167467] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:39:05.707 INFO 2458 --- [prometheus-minute-schedule-1,cbe0cdfcd6e72909,8449074af983b33f] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:05.714 INFO 2458 --- [prometheus-minute-schedule-1,cbe0cdfcd6e72909,8449074af983b33f] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:06.166 INFO 2458 --- [ocp-schedules-8,74cc20836cbffdbd,5b1607def426b139] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:06.170 INFO 2458 --- [ocp-schedules-8,74cc20836cbffdbd,5b1607def426b139] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:39:06.692 WARN 2458 --- [ocp-async-10,bc200fc9510d8a9d,bee1d703fc44327d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:06.692 INFO 2458 --- [ocp-async-5,41d2fd8ea7442e77,e075769ed6b903ae] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:06.748 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:06.749 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:07.304 INFO 2458 --- [ocp-schedules-27,2f5154ef6207dd08,5a55032736e85540] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:07.951 INFO 2458 --- [ocp-schedules-29,5ab26286b2d32060,9af811274a091b01] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:39:07.959 INFO 2458 --- [ocp-schedules-29,5ab26286b2d32060,9af811274a091b01] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:07.959 INFO 2458 --- [ocp-schedules-29,5ab26286b2d32060,9af811274a091b01] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:08.545 INFO 2458 --- [ocp-async-7,93218b0f8b52c992,a9733a4b62a9b70e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:08.692 WARN 2458 --- [ocp-async-1,550140505601a72c,f216f596578f208a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:08.692 INFO 2458 --- [ocp-async-8,a49dcb7791db8f6d,68e771e4a33b562d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:08.707 INFO 2458 --- [prometheus-minute-schedule-1,e5dff570293a6e28,13a48f7d99af316d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:08.713 INFO 2458 --- [prometheus-minute-schedule-1,e5dff570293a6e28,13a48f7d99af316d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:09.749 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:09.750 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:10.012 INFO 2458 --- [ocp-schedules-20,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:39:10.014 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:39:10.072 INFO 2458 --- [Check obproxy1,86124ecee75462a8,a2d76fca0afbff3b] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:39:10.072 INFO 2458 --- [Collect obproxy1,4bbbe7d9e9f648b9,d3127c85ac0b1340] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:39:10.078 INFO 2458 --- [Check obproxy1,86124ecee75462a8,a2d76fca0afbff3b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:39:10.079 INFO 2458 --- [Collect obproxy1,4bbbe7d9e9f648b9,d3127c85ac0b1340] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:39:10.106 INFO 2458 --- [Check obproxy1,86124ecee75462a8,a2d76fca0afbff3b] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:39:10.108 INFO 2458 --- [Collect obproxy1,4bbbe7d9e9f648b9,d3127c85ac0b1340] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:39:10.110 INFO 2458 --- [Check obproxy1,86124ecee75462a8,a2d76fca0afbff3b] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:39:10.110 INFO 2458 --- [Check obproxy1,86124ecee75462a8,a2d76fca0afbff3b] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:39:10.110 INFO 2458 --- [Collect obproxy1,4bbbe7d9e9f648b9,d3127c85ac0b1340] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:39:10.113 INFO 2458 --- [Collect obproxy1,4bbbe7d9e9f648b9,d3127c85ac0b1340] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:39:10.188 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:39:10.692 WARN 2458 --- [ocp-async-8,f26eed4c469c3e6f,d97c59c6951d9769] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:10.692 INFO 2458 --- [ocp-async-9,a6e0916e0ac8cbd4,5eb359328994094f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:11.507 ERROR 2458 --- [metric-archive-second-3,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134505683) Duplicate entry '835014129065861120-1761536220' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134505683) Duplicate entry '835014129065861120-1761536220' for key 'PRIMARY' 2025-10-27 11:39:11.556 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=1000, affectRows=1000 2025-10-27 11:39:11.560 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=999, validCount=995 2025-10-27 11:39:11.578 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134602000) Duplicate entry '835370602992181248-1761536220' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134602000) Duplicate entry '835370602992181248-1761536220' for key 'PRIMARY' 2025-10-27 11:39:11.581 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=1000, affectRows=1000 2025-10-27 11:39:11.586 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=994, validCount=992 2025-10-27 11:39:11.643 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=1000, affectRows=1000 2025-10-27 11:39:11.650 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=996, validCount=992 2025-10-27 11:39:11.667 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=80, affectRows=80 2025-10-27 11:39:11.668 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=78, validCount=78 2025-10-27 11:39:11.707 INFO 2458 --- [prometheus-minute-schedule-1,b1e1aef6ea01daca,5ac50b1befa26c15] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:11.716 INFO 2458 --- [prometheus-minute-schedule-1,b1e1aef6ea01daca,5ac50b1befa26c15] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:11.719 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:11.718298+08:00, lastProcessTime=2025-10-27T11:39:01.715358+08:00 2025-10-27 11:39:11.970 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:11.718298+08:00 2025-10-27 11:39:12.430 INFO 2458 --- [metric-archive-async-4,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536280, nodeCount=821, affectRows=821 2025-10-27 11:39:12.692 WARN 2458 --- [ocp-async-7,75347a0bf8baa9db,f3030f9a39e4aba6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:12.692 INFO 2458 --- [ocp-async-5,4a0784eb8f857949,a992ada87948cec4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:12.750 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:12.751 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:12.797 INFO 2458 --- [ocp-schedules-15,e78a4454475aa077,423d2405d1b2040e] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:12.842 WARN 2458 --- [scanner-112225816-1,e78a4454475aa077,57a613d2b63183ac] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:12.987 INFO 2458 --- [alarm-detect-12,e78a4454475aa077,1c5f4375136f1764] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536325 ~ 1761536342 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536340, value=72.481)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536340, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536340=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:14.692 WARN 2458 --- [ocp-async-3,07db864d0feb13ef,016fe3c2b6331c96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:14.692 INFO 2458 --- [ocp-async-1,27d8cdd1773466af,275b630aa7ccd7c8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:14.707 INFO 2458 --- [prometheus-minute-schedule-1,21d5e62d743e9557,6cc940d5882618c9] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:14.713 INFO 2458 --- [prometheus-minute-schedule-1,21d5e62d743e9557,6cc940d5882618c9] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:15.751 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:15.753 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:16.187 INFO 2458 --- [ocp-schedules-29,0f4839814a652e45,57c4a689f6504dc4] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:16.191 INFO 2458 --- [ocp-schedules-29,0f4839814a652e45,57c4a689f6504dc4] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:39:16.305 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=1000, affectRows=1000 2025-10-27 11:39:16.309 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=1000, validCount=986 2025-10-27 11:39:16.368 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536280, nodeCount=1000, affectRows=1000 2025-10-27 11:39:16.372 INFO 2458 --- [metric-archive-second-3,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536280, nodeCount=996, validCount=994 2025-10-27 11:39:16.662 INFO 2458 --- [ocp-schedules-11,b4276bd6f9e45aea,36b53b11487c1f13] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:39:16.662+08:00, clusters=[1, 2] 2025-10-27 11:39:16.665 INFO 2458 --- [ocp-schedules-11,b4276bd6f9e45aea,36b53b11487c1f13] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:39:16.667 INFO 2458 --- [ocp-schedules-11,b4276bd6f9e45aea,36b53b11487c1f13] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:39:16.692 WARN 2458 --- [ocp-async-6,d09eae947f795a5c,e2747aadafe4a510] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:16.692 INFO 2458 --- [ocp-async-8,ebdd3f2ebdecad21,c7219d79c62b3876] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:17.311 INFO 2458 --- [ocp-schedules-12,c90d6c8b704b074d,1a82c470ae5b5ff0] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:17.707 INFO 2458 --- [prometheus-minute-schedule-1,d26461ab53181ecb,9327ef96a9e3e0e4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:17.714 INFO 2458 --- [prometheus-minute-schedule-1,d26461ab53181ecb,9327ef96a9e3e0e4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:17.950 INFO 2458 --- [ocp-schedules-6,3b8e63b243ea3c84,2d9c94d912c9003a] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:39:17.957 INFO 2458 --- [ocp-schedules-6,3b8e63b243ea3c84,2d9c94d912c9003a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:18.046 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:39:18.046 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.050 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:39:18.062 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.063 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.088 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.089 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.093 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:39:18.093 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.096 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:39:18.116 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.117 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.117 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.118 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.120 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.120 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.146 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.147 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.148 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.149 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.214 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.215 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:39:18.217 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.218 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.218 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.219 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.251 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.251 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.251 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.252 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.252 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.253 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.254 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.254 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.255 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.256 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.257 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.258 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.258 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.259 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.260 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:39:18.270 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:39:18.280 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.280 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:39:18.283 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.284 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:39:18.320 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.320 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:39:18.326 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:18.326 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.327 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:39:18.338 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.338 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:39:18.338 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.338 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:39:18.343 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:39:18.343 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:39:18.352 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.353 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:39:18.355 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.356 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:39:18.385 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.386 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:39:18.412 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.413 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:39:18.458 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.459 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.460 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.460 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.461 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.462 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.463 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.464 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.464 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.465 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.466 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.467 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.468 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.468 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.469 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.470 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.470 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.471 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.472 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.472 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:39:18.473 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.474 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:39:18.474 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.475 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:39:18.476 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:39:18.479 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:39:18.546 INFO 2458 --- [ocp-async-1,c16af223218959cc,a7198b5ba63dbd5c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:18.567 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:18.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.568 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:39:18.596 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:39:18.602 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:39:18.602 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:39:18.602 INFO 2458 --- [ocp-schedules-6,3b8e63b243ea3c84,2d9c94d912c9003a] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:39:18.692 WARN 2458 --- [ocp-async-7,2bee2cdbe94f9389,237e864d248d4274] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:18.692 INFO 2458 --- [ocp-async-10,eddefc6924987bea,3a00dedc4257f1de] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:18.753 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:18.753 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:18.848 INFO 2458 --- [ocp-schedules-16,ea0c4f6643e21023,3c2a28645fe9c165] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:39:18.848 INFO 2458 --- [ocp-schedules-16,ea0c4f6643e21023,3c2a28645fe9c165] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:39:18.907 INFO 2458 --- [Collect all tenants major compactions1,ea0c4f6643e21023,c640878257e5b462] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:39:18.907 INFO 2458 --- [Collect all tenants major compactions3,ea0c4f6643e21023,853a4a110b7f4aa8] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:39:18.907 INFO 2458 --- [Collect all tenants major compactions4,ea0c4f6643e21023,26460cac03730955] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:39:18.908 INFO 2458 --- [Collect all tenants major compactions6,ea0c4f6643e21023,7af69713b27efe5e] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:39:18.908 INFO 2458 --- [Collect all tenants major compactions5,ea0c4f6643e21023,aba27844a61a0c0f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:39:18.910 INFO 2458 --- [Collect all tenants major compactions2,ea0c4f6643e21023,4bee60c542886133] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:39:18.944 INFO 2458 --- [Collect all tenants major compactions5,ea0c4f6643e21023,aba27844a61a0c0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.944 INFO 2458 --- [Collect all tenants major compactions6,ea0c4f6643e21023,7af69713b27efe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.945 INFO 2458 --- [Collect all tenants major compactions5,ea0c4f6643e21023,aba27844a61a0c0f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:39:18.946 INFO 2458 --- [Collect all tenants major compactions6,ea0c4f6643e21023,7af69713b27efe5e] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:39:18.958 INFO 2458 --- [Collect all tenants major compactions6,ea0c4f6643e21023,7af69713b27efe5e] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:39:18.959 INFO 2458 --- [Collect all tenants major compactions1,ea0c4f6643e21023,c640878257e5b462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.959 INFO 2458 --- [Collect all tenants major compactions4,ea0c4f6643e21023,26460cac03730955] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.959 INFO 2458 --- [Collect all tenants major compactions2,ea0c4f6643e21023,4bee60c542886133] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.961 INFO 2458 --- [Collect all tenants major compactions1,ea0c4f6643e21023,c640878257e5b462] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:39:18.961 INFO 2458 --- [Collect all tenants major compactions4,ea0c4f6643e21023,26460cac03730955] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:39:18.961 INFO 2458 --- [Collect all tenants major compactions2,ea0c4f6643e21023,4bee60c542886133] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:39:18.964 INFO 2458 --- [Collect all tenants major compactions3,ea0c4f6643e21023,853a4a110b7f4aa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:18.964 INFO 2458 --- [Collect all tenants major compactions4,ea0c4f6643e21023,26460cac03730955] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:39:18.964 INFO 2458 --- [Collect all tenants major compactions1,ea0c4f6643e21023,c640878257e5b462] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:39:18.965 INFO 2458 --- [Collect all tenants major compactions5,ea0c4f6643e21023,aba27844a61a0c0f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:39:18.965 INFO 2458 --- [Collect all tenants major compactions2,ea0c4f6643e21023,4bee60c542886133] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:39:18.966 INFO 2458 --- [Collect all tenants major compactions3,ea0c4f6643e21023,853a4a110b7f4aa8] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:39:18.969 INFO 2458 --- [Collect all tenants major compactions3,ea0c4f6643e21023,853a4a110b7f4aa8] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:39:18.970 INFO 2458 --- [ocp-schedules-16,ea0c4f6643e21023,3c2a28645fe9c165] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:39:20.693 WARN 2458 --- [ocp-async-2,54ca1676e23c40c4,c71dc61c2a746343] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:20.693 INFO 2458 --- [ocp-async-5,b97b07ded00810e9,7d27b317d4183bd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:20.707 INFO 2458 --- [prometheus-minute-schedule-1,177454717c0b7e14,a9ef28c5817acd07] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:20.714 INFO 2458 --- [prometheus-minute-schedule-1,177454717c0b7e14,a9ef28c5817acd07] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:20.996 INFO 2458 --- [ocp-schedules-2,c34557c7b6a3d4f3,d832947e13f40378] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:39:21.411 INFO 2458 --- [ocp-schedules-24,5aa978d0d50c2b09,5efafeb249dc31a6] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:21.411 INFO 2458 --- [ocp-schedules-13,964eb65143be8e40,9d08283188defd15] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:39:21.412 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:39:21.413 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:39:21.413 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:39:21.414 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:39:21.426 INFO 2458 --- [ocp-schedules-6,40528924a375d504,2e9977732fd59bb6] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:21.427 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:21.427 INFO 2458 --- [host-check-clockdiff1,40528924a375d504,76d3356450808353] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:39:21.427 INFO 2458 --- [host-check-clockdiff2,40528924a375d504,549a5247083942b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:39:21.427 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:21.427 INFO 2458 --- [host-check-clockdiff3,40528924a375d504,4d69610232326e92] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:39:21.428 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:21.428 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:21.428 INFO 2458 --- [host-check-clockdiff5,40528924a375d504,5dd014afc043033b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:39:21.428 INFO 2458 --- [host-check-clockdiff4,40528924a375d504,d16350fdb18870fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:39:21.429 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:21.429 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:21.432 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:21.432 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:21.432 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:21.433 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:21.433 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:21.433 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:21.433 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:21.433 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:21.434 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:21.442 INFO 2458 --- [ocp-schedules-23,ff4123e447992d07,224ded3baaacb7e7] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:39:21.442 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:21.442 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:21.443 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:21.444 INFO 2458 --- [host-check-clockdiff1,40528924a375d504,76d3356450808353] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536361 0 0 , error= 2025-10-27 11:39:21.444 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:21.449 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:21.450 INFO 2458 --- [host-check-clockdiff3,40528924a375d504,4d69610232326e92] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536361 0 0 , error= 2025-10-27 11:39:21.454 INFO 2458 --- [host-check-clockdiff2,40528924a375d504,549a5247083942b5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536361 -1 -1 , error= 2025-10-27 11:39:21.545 INFO 2458 --- [ocp-schedules-15,b5de1434edef9412,d0b58277a939f892] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:39:21.549 INFO 2458 --- [ocp-schedules-26,81bb5b2e3bcca467,ea7a7ffa80b5c61b] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:39:21.551 INFO 2458 --- [ocp-schedules-15,b5de1434edef9412,d0b58277a939f892] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:21.555 INFO 2458 --- [ocp-schedules-26,81bb5b2e3bcca467,ea7a7ffa80b5c61b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:21.555 INFO 2458 --- [ocp-schedules-22,31d84b085e4fc1fe,dafc2dbf6977f3aa] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:39:21.562 INFO 2458 --- [ocp-schedules-22,31d84b085e4fc1fe,dafc2dbf6977f3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:21.568 INFO 2458 --- [ocp-schedules-5,f2c7cbfe659f52da,63edb931f5c4dae8] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:39:21.568 INFO 2458 --- [ocp-schedules-26,81bb5b2e3bcca467,ea7a7ffa80b5c61b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:39:21.603 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:39:21.607 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:39:21.608 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:39:21.608 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:39:21.608 INFO 2458 --- [host-check-agent-status1,964eb65143be8e40,70c4ec489a995a57] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:39:21.609 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:39:21.609031542+08:00 2025-10-27 11:39:21.613 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:39:21.609+08:00 2025-10-27 11:39:21.620 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:39:21.622 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:39:21.622383545+08:00 2025-10-27 11:39:21.623 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:39:21.622+08:00 2025-10-27 11:39:21.625 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:39:21.609+08:00 and status AVAILABLE 2025-10-27 11:39:21.632 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:39:21.638 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:39:21.622+08:00 and status AVAILABLE 2025-10-27 11:39:21.641 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.642 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:21.643 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:39:21.646 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:39:21.647 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:39:21.647 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.648 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:39:21.647492608+08:00 2025-10-27 11:39:21.648 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:21.649 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:39:21.648+08:00 2025-10-27 11:39:21.652 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:39:21.653 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:39:21.653 INFO 2458 --- [host-check-agent-status2,964eb65143be8e40,082248997b7219b5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:39:21.653 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:39:21.653 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:39:21.654 INFO 2458 --- [ocp-schedules-15,b5de1434edef9412,d0b58277a939f892] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:39:21.656 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:39:21.656 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:39:21.656681702+08:00 2025-10-27 11:39:21.657 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:39:21.657+08:00 2025-10-27 11:39:21.659 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:39:21.648+08:00 and status AVAILABLE 2025-10-27 11:39:21.661 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:21.664 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:21.666 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:39:21.657+08:00 and status AVAILABLE 2025-10-27 11:39:21.669 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.669 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:39:21.670 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:21.672 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:39:21.680 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:21.683 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:39:21.683 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:21.684 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:39:21.682758199+08:00 2025-10-27 11:39:21.684 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:39:21.684+08:00 2025-10-27 11:39:21.686 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:39:21.689 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:39:21.689 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:39:21.689 INFO 2458 --- [host-check-agent-status5,964eb65143be8e40,81aa669dcd7db679] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:39:21.689 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:39:21.692 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:39:21.684+08:00 and status AVAILABLE 2025-10-27 11:39:21.692 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:39:21.692 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:39:21.692 INFO 2458 --- [host-check-agent-status3,964eb65143be8e40,0338d2769a6cd2f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:39:21.694 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:39:21.696 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:21.697 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:39:21.697 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:39:21.697 INFO 2458 --- [host-check-agent-status4,964eb65143be8e40,82faa79ebe57cdb2] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:39:21.699 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:21.699 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:21.704 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:39:21.704 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:21.713 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.713 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:21.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:21.715947+08:00, lastProcessTime=2025-10-27T11:39:11.718298+08:00 2025-10-27 11:39:21.732 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:21.732 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:21.735 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:21.749 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:39:21.751 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:21.752 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.752 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:21.753 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:39:21.754 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:21.755 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:21.768 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:39:21.770 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:21.774 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:21.791 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:39:21.796 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:39:21.807 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:39:21.807 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:39:21.808 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:21.809 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.810 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:39:21.811 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:21.822 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:21.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.823 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:21.824 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.828 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:39:21.829 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:21.832 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:39:21.836 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:39:21.837 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:39:21.857 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:21.862 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:21.869 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:21.869 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:21.872 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:39:21.878 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:39:21.881 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:39:21.888 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:21.889 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:21.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:21.909 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:39:21.916 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:39:21.916 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:39:21.919 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.920 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:21.922 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.926 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:39:21.927 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:21.933 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:21.937 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:39:21.945 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:21.946 INFO 2458 --- [ocp-schedules-30,f70be862a31b901a,1f560545df1b724a] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:39:21.946 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:21.947 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:21.948 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:39:21.952 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:21.954 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:39:21.960 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:39:21.962 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:21.969 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:39:21.972 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:39:21.973 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:39:21.973 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:21.975 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:21.976 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:21.979 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:39:22.000 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:39:22.002 INFO 2458 --- [host-check-host-status4,5aa978d0d50c2b09,1883cb1342bbdc8b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:39:22.015 INFO 2458 --- [host-check-host-status5,5aa978d0d50c2b09,0024bfba6691fa1b] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:39:22.015 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:22.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:22.026 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:22.031 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:22.038 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:21.715947+08:00 2025-10-27 11:39:22.048 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:22.051 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:39:22.052 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:39:22.059 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:39:22.069 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:39:22.075 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:22.075 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:22.083 INFO 2458 --- [host-check-host-status1,5aa978d0d50c2b09,487547ac9f9a3fdc] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:39:22.083 INFO 2458 --- [host-check-host-status3,5aa978d0d50c2b09,d4db84761bf77ce3] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:39:22.098 INFO 2458 --- [host-check-host-status2,5aa978d0d50c2b09,5362340b549b4d2f] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:39:22.102 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:39:22.103 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.103 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:39:22.105 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.106 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:39:22.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:22.115 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:22.123 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:39:22.149 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:22.150 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.151 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:39:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:22.152 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:22.174 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:39:22.174 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:39:22.189 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:22.231 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:22.231 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.232 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:22.236 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:39:22.246 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:39:22.255 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:22.259 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:22.309 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:39:22.371 INFO 2458 --- [host-check-clockdiff4,40528924a375d504,d16350fdb18870fa] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536362 -1 -1 , error= 2025-10-27 11:39:22.381 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:39:22.381 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.391 INFO 2458 --- [host-check-clockdiff5,40528924a375d504,5dd014afc043033b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536362 0 0 , error= 2025-10-27 11:39:22.391 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:39:22.393 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.394 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:39:22.428 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:39:22.470 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:22.471 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:22.472 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:39:22.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:39:22.510 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:39:22.510 INFO 2458 --- [ocp-schedules-22,31d84b085e4fc1fe,dafc2dbf6977f3aa] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:39:22.693 WARN 2458 --- [ocp-async-5,f6e357b63cf8d374,0793ba7cbd1242d4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:22.693 INFO 2458 --- [ocp-async-4,2c51c501557c4ba5,d68e51fe866f97f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:22.992 INFO 2458 --- [ocp-schedules-23,145aca46bbdc8bef,9d92964821e4dc06] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:23.031 WARN 2458 --- [scanner-112226111-1,145aca46bbdc8bef,e9e210a98af0b6d3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:23.056 INFO 2458 --- [alarm-detect-5,145aca46bbdc8bef,d94171eb3c0efd27] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536325 ~ 1761536353 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536340, value=1.563)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.563, values={1761536340=1.563}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:23.707 INFO 2458 --- [prometheus-minute-schedule-1,8f518e1b601065fe,a96c6176de2e1f8d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:23.716 INFO 2458 --- [prometheus-minute-schedule-1,8f518e1b601065fe,a96c6176de2e1f8d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:24.693 INFO 2458 --- [ocp-async-7,988af4ec75d3088e,9b09ca70f0653465] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:24.693 WARN 2458 --- [ocp-async-3,c49c10a3baab96ac,55c274aa2579a444] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:24.755 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:24.756 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:26.200 INFO 2458 --- [ocp-schedules-20,860bdfc2de192bd6,6d0ee65a6e992330] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:26.204 INFO 2458 --- [ocp-schedules-20,860bdfc2de192bd6,6d0ee65a6e992330] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:39:26.693 WARN 2458 --- [ocp-async-7,74802a7d5d8c7b8d,f538ffbdf18d657c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:26.693 INFO 2458 --- [ocp-async-7,20b1390fda510484,4faec2dccdcb2bfc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:26.707 INFO 2458 --- [prometheus-minute-schedule-1,b196ce31c891caf9,2587d24017b04ad2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:26.713 INFO 2458 --- [prometheus-minute-schedule-1,b196ce31c891caf9,2587d24017b04ad2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:26.894 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /api/v2/software-packages?null, client=10.40.198.13, traceId=ffb81588d0a4e3e1, method=SuccessResponse com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(MultipartFile), args=] 2025-10-27 11:39:26.940 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] .o.o.s.o.i.LocalObjectStorageServiceImpl : Init saving object, creatorId=100, bucket=sys-package, name=obbinlog-ce-4.2.5-100000042025052216.el7.x86_64.rpm, objectId=14 2025-10-27 11:39:27.317 INFO 2458 --- [ocp-schedules-17,81bb0c253eb1866c,ade6811643998ebe] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:27.756 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:27.758 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:27.962 INFO 2458 --- [ocp-schedules-16,00a9d379ae98022c,5c2ef24a1bb2de81] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:39:27.970 INFO 2458 --- [ocp-schedules-16,00a9d379ae98022c,5c2ef24a1bb2de81] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:27.970 INFO 2458 --- [ocp-schedules-16,00a9d379ae98022c,5c2ef24a1bb2de81] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:28.546 INFO 2458 --- [ocp-async-7,2041f83f54d1cc30,68403a336c114774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:28.693 WARN 2458 --- [ocp-async-1,a9cfbb15bd3a0d23,5f3437dc08d45341] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:28.693 INFO 2458 --- [ocp-async-8,da22c6231013eb4b,34df847931385a03] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:29.108 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:39:29.108 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:39:29.114 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:39:29.114 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839265989532524544 2025-10-27 11:39:29.118 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:39:29.118 INFO 2458 --- [ocp-schedules-20,ec6c06c233170f1d,c7e385361374d41c] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:39:29.564 INFO 2458 --- [ocp-schedules-12,41d90e4bb9e34254,6f68e0585caf7694] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:39:29.571 INFO 2458 --- [ocp-schedules-12,41d90e4bb9e34254,6f68e0585caf7694] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:39:29.571 INFO 2458 --- [ocp-schedules-12,41d90e4bb9e34254,6f68e0585caf7694] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:39:29.707 INFO 2458 --- [prometheus-minute-schedule-1,70e3983736cc4a78,e070207f11d864b5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:29.716 INFO 2458 --- [prometheus-minute-schedule-1,70e3983736cc4a78,e070207f11d864b5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:29.785 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] .o.o.s.o.i.LocalObjectStorageServiceImpl : Unexpected exception when saving. com.oceanbase.ocp.core.exception.OcpException: [OCP OcpException]: status=503 SERVICE_UNAVAILABLE, errorCode=COMMON_TIMEOUT, args= at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator.saveLocalFileToDbByBlock(ObjectBlockOperator.java:149) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator.saveObjectBlock(ObjectBlockOperator.java:59) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$FastClassBySpringCGLIB$$584a6a26.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$EnhancerBySpringCGLIB$$b3c17fef.saveObjectBlock() at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.doPutObject(LocalObjectStorageServiceImpl.java:304) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.lambda$putExternalObject$0(LocalObjectStorageServiceImpl.java:112) at com.oceanbase.ocp.storage.object.util.ObjectStorageExecutor.concurrentSafeExecute(ObjectStorageExecutor.java:56) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.putExternalObject(LocalObjectStorageServiceImpl.java:111) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$FastClassBySpringCGLIB$$31292e74.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$EnhancerBySpringCGLIB$$c71bed99.putExternalObject() at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy.putRpm(ObjectStorageServiceProxy.java:71) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$FastClassBySpringCGLIB$$e9646277.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$EnhancerBySpringCGLIB$$3f40a382.putRpm() at com.oceanbase.ocp.storage.softwarepackage.SoftwarePackageServiceImpl.storePackage(SoftwarePackageServiceImpl.java:127) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(SoftwarePackageController.java:71) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$FastClassBySpringCGLIB$$38e03a7f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:138) at sun.reflect.GeneratedMethodAccessor340.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$EnhancerBySpringCGLIB$$d692721d.uploadPackage() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) 2025-10-27 11:39:29.785 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] .o.o.s.o.i.LocalObjectStorageServiceImpl : Clean failed upload record, objectId=14 2025-10-27 11:39:30.693 INFO 2458 --- [ocp-async-4,a518afb1b3aa43d3,3586c574046d681a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:30.693 WARN 2458 --- [ocp-async-9,76f21f13a6b67766,a2920fb7477dba52] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:30.758 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:30.759 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:31.716351+08:00, lastProcessTime=2025-10-27T11:39:21.715947+08:00 2025-10-27 11:39:31.949 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:31.716351+08:00 2025-10-27 11:39:32.693 WARN 2458 --- [ocp-async-7,9a1fb2368c0a7674,4df8a0a53093ade3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:32.693 INFO 2458 --- [ocp-async-4,bb4e54b5703e27f6,18a51095b6d4d687] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:32.707 INFO 2458 --- [prometheus-minute-schedule-1,232e6107301a80e7,aeb3f2b8b53c6ac1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:32.715 INFO 2458 --- [prometheus-minute-schedule-1,232e6107301a80e7,aeb3f2b8b53c6ac1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:33.111 INFO 2458 --- [ocp-schedules-18,3974dce2f81e6c3e,5bad48db6f2bdae2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:33.145 WARN 2458 --- [scanner-112226254-1,3974dce2f81e6c3e,5fb3fe6b0174d47b] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:33.214 INFO 2458 --- [alarm-detect-19,3974dce2f81e6c3e,156df2d763296485] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536340 ~ 1761536363 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536355, value=1.531)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.531, values={1761536355=1.531}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:33.214 INFO 2458 --- [alarm-detect-19,3974dce2f81e6c3e,156df2d763296485] c.o.o.a.c.detect.AlarmRuleDetectState : checkPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, currentState=Firing 2025-10-27 11:39:33.215 INFO 2458 --- [alarm-detect-19,3974dce2f81e6c3e,156df2d763296485] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_host_load1_per_cpu_over_threshold, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:host=10.40.65.146-3] 2025-10-27 11:39:33.290 INFO 2458 --- [alarm-detect-2,3974dce2f81e6c3e,85229c743dd99149] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536340 ~ 1761536363 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536355, value=67.002)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536355, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536355=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:33.759 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:34.036 INFO 2458 --- [ocp-schedules-17,8714cb4efece3906,84821be99ebd27f6] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:39:34.036 INFO 2458 --- [ocp-schedules-17,8714cb4efece3906,84821be99ebd27f6] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:39:34.657 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obQueryCache 2025-10-27 11:39:34.657 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.ocp.core.cache.helper.ObQueryCache : Init ob cache. 2025-10-27 11:39:34.694 WARN 2458 --- [ocp-async-10,af4abd31f6bb2ffb,3e1ec7dc81f26ab7] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:34.694 INFO 2458 --- [ocp-async-3,53cbb97d36d31f62,4574730c9c03b410] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:34.704 WARN 2458 --- [ocp-async-3,53cbb97d36d31f62,4574730c9c03b410] c.o.o.s.common.DistributedLockAspect : Acquire lock fail, another request is processing, key=AlarmEventProcessForSelfcure 2025-10-27 11:39:35.707 INFO 2458 --- [prometheus-minute-schedule-1,9cdf5825bf7378e0,35b85e86b83bfa69] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:36.298 WARN 2458 --- [ocp-schedules-18,3974dce2f81e6c3e,5bad48db6f2bdae2] c.o.ocp.alarm.core.task.AlarmDetectTask : Specified waiting time elapses before detect return. 2025-10-27 11:39:36.600 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:36.605 INFO 2458 --- [prometheus-minute-schedule-1,9cdf5825bf7378e0,35b85e86b83bfa69] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:36.605 INFO 2458 --- [prometheus-minute-schedule-1,9cdf5825bf7378e0,35b85e86b83bfa69] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1761536375707, count=14, elapsed=898 2025-10-27 11:39:36.694 INFO 2458 --- [ocp-async-3,7134c10650befb3d,9f8c3b031d842c9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:36.694 WARN 2458 --- [ocp-async-7,c68cc2ff869e64fb,32477b104c04362c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:36.704 WARN 2458 --- [ocp-async-3,7134c10650befb3d,9f8c3b031d842c9c] c.o.o.s.common.DistributedLockAspect : Acquire lock fail, another request is processing, key=AlarmEventProcessForSelfcure 2025-10-27 11:39:37.324 INFO 2458 --- [ocp-schedules-23,bbf79d9f8f7e0778,a20bc5c86d7c4ddc] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:37.708 INFO 2458 --- [prometheus-cache-monitor1,1fef6ce3a8b0d7b0,1e4dc2683f7b0ce5] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20324,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:39:37.708 INFO 2458 --- [prometheus-cache-monitor1,1fef6ce3a8b0d7b0,1e4dc2683f7b0ce5] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:39:38.546 INFO 2458 --- [ocp-async-5,e910055d8f2fe87e,82821ddaff9f2e9a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:38.694 WARN 2458 --- [ocp-async-10,1d8bae574f96195b,43884a94b2493321] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:38.694 INFO 2458 --- [ocp-async-3,e1f344df94f8c765,b80e7ae86c77b616] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:38.704 WARN 2458 --- [ocp-async-3,e1f344df94f8c765,b80e7ae86c77b616] c.o.o.s.common.DistributedLockAspect : Acquire lock fail, another request is processing, key=AlarmEventProcessForSelfcure 2025-10-27 11:39:38.707 INFO 2458 --- [prometheus-minute-schedule-1,06a49893787660d9,c34cf60974df9bcc] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:38.952 INFO 2458 --- [ocp-schedules-17,eb809457666d0a3c,b28e11d1b671a64c] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:39:39.600 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:39.660 INFO 2458 --- [alarm-detect-19,3974dce2f81e6c3e,156df2d763296485] c.o.o.alarm.service.OcpAlarmServiceImpl : create alarm event Done, id=354 2025-10-27 11:39:39.706 WARN 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] o.h.engine.jdbc.spi.SqlExceptionHelper : SQL Error: 4012, SQLState: HY000 2025-10-27 11:39:39.706 ERROR 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] o.h.engine.jdbc.spi.SqlExceptionHelper : (conn=134538047) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. 2025-10-27 11:39:39.721 INFO 2458 --- [ocp-schedules-17,eb809457666d0a3c,b28e11d1b671a64c] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:39.721 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:39:39.721 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:39:39.721 INFO 2458 --- [ocp-schedules-17,eb809457666d0a3c,b28e11d1b671a64c] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:39:39.728 ERROR 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] c.o.o.s.c.trace.RequestTracingAspect : Unexpected API Error: [POST /api/v2/software-packages client=10.40.198.13, traceId=ffb81588d0a4e3e1, args=, duration=42889 ms] org.springframework.dao.DataAccessResourceFailureException: could not execute statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551) at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61) at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:243) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy403.deleteByObjectId(Unknown Source) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator.deleteByObjectId(ObjectBlockOperator.java:85) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$FastClassBySpringCGLIB$$584a6a26.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$EnhancerBySpringCGLIB$$b3c17fef.deleteByObjectId() at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.lambda$cleanFailedRecord$8(LocalObjectStorageServiceImpl.java:348) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.cleanFailedRecord(LocalObjectStorageServiceImpl.java:347) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.doPutObject(LocalObjectStorageServiceImpl.java:313) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.lambda$putExternalObject$0(LocalObjectStorageServiceImpl.java:112) at com.oceanbase.ocp.storage.object.util.ObjectStorageExecutor.concurrentSafeExecute(ObjectStorageExecutor.java:56) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.putExternalObject(LocalObjectStorageServiceImpl.java:111) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$FastClassBySpringCGLIB$$31292e74.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$EnhancerBySpringCGLIB$$c71bed99.putExternalObject() at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy.putRpm(ObjectStorageServiceProxy.java:71) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$FastClassBySpringCGLIB$$e9646277.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$EnhancerBySpringCGLIB$$3f40a382.putRpm() at com.oceanbase.ocp.storage.softwarepackage.SoftwarePackageServiceImpl.storePackage(SoftwarePackageServiceImpl.java:127) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(SoftwarePackageController.java:71) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$FastClassBySpringCGLIB$$38e03a7f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:138) at sun.reflect.GeneratedMethodAccessor340.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$EnhancerBySpringCGLIB$$d692721d.uploadPackage() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200) at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:80) at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:50) at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:177) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:458) at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:376) at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1483) at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1714) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1696) at org.springframework.data.jpa.repository.query.JpaQueryExecution$ModifyingExecution.doExecute(JpaQueryExecution.java:241) at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:90) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:156) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:144) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:160) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:81) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ... 200 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=134538047) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:205) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ... 224 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 229 common frames omitted Caused by: java.sql.SQLException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 230 common frames omitted 2025-10-27 11:39:39.731 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] c.o.ocp.service.audit.AuditEventService : begin create audit event, param:CreateAuditEventParam(operationEvent=UPLOAD_PACKAGE, eventType=SOFTWARE_PACKAGE_MANAGEMENT, methodSignature=com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage, resourceType=SOFTWARE_PACKAGE, resources=[CreateAuditEventParam.AuditResourceParam(resourceId=null, resourceName=obbinlog-ce-4.2.5-100000042025052216.el7.x86_64.rpm, parentResource=null)], operatorId=100, operatorName=admin, status=FAILED, failedReason={"errorCode":"COM10015","errorMessage":"could not execute statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute statement"}, taskId=null, eventInfo={"file":"obbinlog-ce-4.2.5-100000042025052216.el7.x86_64.rpm"}, executionType=NEW, requestUrl=http://10.40.198.13:24631/api/v2/software-packages, clientIp=10.40.198.13, serverIp=10.40.65.146, startTime=2025-10-27T11:39:26.894+08:00, endTime=2025-10-27T11:39:39.728+08:00, role=null, aliUid=null) 2025-10-27 11:39:39.731 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:39.735 INFO 2458 --- [ocp-schedules-3,58caeab17b522408,d097f9f4aa82f3ec] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:39.737 INFO 2458 --- [prometheus-minute-schedule-1,06a49893787660d9,c34cf60974df9bcc] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:39.737 INFO 2458 --- [prometheus-minute-schedule-1,06a49893787660d9,c34cf60974df9bcc] c.o.ocp.monitor.OcpMonitorManager : Schedule interval metric too slow, from=1761536378707, count=14, elapsed=1030 2025-10-27 11:39:39.741 INFO 2458 --- [ocp-schedules-3,58caeab17b522408,d097f9f4aa82f3ec] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:39:39.749 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.ocp.core.cache.helper.ObQueryCache : Finished init ob cache. 2025-10-27 11:39:39.749 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: binlogQueryCache 2025-10-27 11:39:39.749 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.c.cache.helper.BinlogQueryCache : Init binlog cache. 2025-10-27 11:39:39.775 ERROR 2458 --- [http-nio-0.0.0.0-8081-exec-9,ffb81588d0a4e3e1,f3babe318a5294f5] c.o.o.s.c.e.RestExceptionHandler : Unhandled exception, uri=/api/v2/software-packages, duration=42939 org.springframework.dao.DataAccessResourceFailureException: could not execute statement; nested exception is org.hibernate.exception.JDBCConnectionException: could not execute statement at org.springframework.orm.jpa.vendor.HibernateJpaDialect.convertHibernateAccessException(HibernateJpaDialect.java:255) at org.springframework.orm.jpa.vendor.HibernateJpaDialect.translateExceptionIfPossible(HibernateJpaDialect.java:233) at org.springframework.orm.jpa.AbstractEntityManagerFactoryBean.translateExceptionIfPossible(AbstractEntityManagerFactoryBean.java:551) at org.springframework.dao.support.ChainedPersistenceExceptionTranslator.translateExceptionIfPossible(ChainedPersistenceExceptionTranslator.java:61) at org.springframework.dao.support.DataAccessUtils.translateIfNecessary(DataAccessUtils.java:243) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:152) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.jpa.repository.support.CrudMethodMetadataPostProcessor$CrudMethodMetadataPopulatingMethodInterceptor.invoke(CrudMethodMetadataPostProcessor.java:145) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241) at com.sun.proxy.$Proxy403.deleteByObjectId(Unknown Source) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator.deleteByObjectId(ObjectBlockOperator.java:85) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$FastClassBySpringCGLIB$$584a6a26.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.operator.ObjectBlockOperator$$EnhancerBySpringCGLIB$$b3c17fef.deleteByObjectId() at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.lambda$cleanFailedRecord$8(LocalObjectStorageServiceImpl.java:348) at org.springframework.transaction.support.TransactionOperations.lambda$executeWithoutResult$0(TransactionOperations.java:68) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at org.springframework.transaction.support.TransactionOperations.executeWithoutResult(TransactionOperations.java:67) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.cleanFailedRecord(LocalObjectStorageServiceImpl.java:347) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.doPutObject(LocalObjectStorageServiceImpl.java:313) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.lambda$putExternalObject$0(LocalObjectStorageServiceImpl.java:112) at com.oceanbase.ocp.storage.object.util.ObjectStorageExecutor.concurrentSafeExecute(ObjectStorageExecutor.java:56) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl.putExternalObject(LocalObjectStorageServiceImpl.java:111) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$FastClassBySpringCGLIB$$31292e74.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.impl.LocalObjectStorageServiceImpl$$EnhancerBySpringCGLIB$$c71bed99.putExternalObject() at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy.putRpm(ObjectStorageServiceProxy.java:71) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$FastClassBySpringCGLIB$$e9646277.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy.invokeMethod(CglibAopProxy.java:386) at org.springframework.aop.framework.CglibAopProxy.access$000(CglibAopProxy.java:85) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:703) at com.oceanbase.ocp.storage.object.ObjectStorageServiceProxy$$EnhancerBySpringCGLIB$$3f40a382.putRpm() at com.oceanbase.ocp.storage.softwarepackage.SoftwarePackageServiceImpl.storePackage(SoftwarePackageServiceImpl.java:127) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController.uploadPackage(SoftwarePackageController.java:71) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$FastClassBySpringCGLIB$$38e03a7f.invoke() at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:792) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.AspectJAfterThrowingAdvice.invoke(AspectJAfterThrowingAdvice.java:64) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.AfterReturningAdviceInterceptor.invoke(AfterReturningAdviceInterceptor.java:57) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.adapter.MethodBeforeAdviceInterceptor.invoke(MethodBeforeAdviceInterceptor.java:58) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) at com.oceanbase.ocp.server.common.aspect.OperationEventAspect.aroundAuditEvent(OperationEventAspect.java:138) at sun.reflect.GeneratedMethodAccessor340.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:762) at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:707) at com.oceanbase.ocp.server.common.controller.SoftwarePackageController$$EnhancerBySpringCGLIB$$d692721d.uploadPackage() at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:898) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:809) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) at javax.servlet.http.HttpServlet.service(HttpServlet.java:555) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) at javax.servlet.http.HttpServlet.service(HttpServlet.java:623) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:199) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.servlet.resource.ResourceUrlEncodingFilter.doFilter(ResourceUrlEncodingFilter.java:67) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:337) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:115) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:126) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:81) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:149) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.CredentialsExpiredFilter.doFilterInternal(CredentialsExpiredFilter.java:51) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreUsernamePasswordAuthenticationFilter.doFilter(PreUsernamePasswordAuthenticationFilter.java:41) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.PreAuthenticationFilter.doFilterInternal(PreAuthenticationFilter.java:110) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:223) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:217) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at com.oceanbase.ocp.server.common.config.security.filter.TestLoginAuthenticationFilter.doFilterInternal(TestLoginAuthenticationFilter.java:33) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.saml2.provider.service.servlet.filter.Saml2WebSsoAuthenticationRequestFilter.doFilterInternal(Saml2WebSsoAuthenticationRequestFilter.java:192) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:178) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:103) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:89) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.csrf.CsrfFilter.doFilterInternal(CsrfFilter.java:132) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:90) at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:75) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:112) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:82) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:55) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.session.DisableEncodeUrlFilter.doFilterInternal(DisableEncodeUrlFilter.java:42) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:346) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:221) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:186) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at com.oceanbase.ocp.server.common.trace.TraceFilter.doFilter(TraceFilter.java:50) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.session.web.http.SessionRepositoryFilter.doFilterInternal(SessionRepositoryFilter.java:141) at org.springframework.session.web.http.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:82) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:354) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:267) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:168) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:346) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:396) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:937) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.lang.Thread.run(Thread.java:750) Caused by: org.hibernate.exception.JDBCConnectionException: could not execute statement at org.hibernate.exception.internal.SQLExceptionTypeDelegate.convert(SQLExceptionTypeDelegate.java:48) at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:37) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113) at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:99) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:200) at org.hibernate.hql.internal.ast.exec.BasicExecutor.doExecute(BasicExecutor.java:80) at org.hibernate.hql.internal.ast.exec.BasicExecutor.execute(BasicExecutor.java:50) at org.hibernate.hql.internal.ast.exec.DeleteExecutor.execute(DeleteExecutor.java:177) at org.hibernate.hql.internal.ast.QueryTranslatorImpl.executeUpdate(QueryTranslatorImpl.java:458) at org.hibernate.engine.query.spi.HQLQueryPlan.performExecuteUpdate(HQLQueryPlan.java:376) at org.hibernate.internal.SessionImpl.executeUpdate(SessionImpl.java:1483) at org.hibernate.query.internal.AbstractProducedQuery.doExecuteUpdate(AbstractProducedQuery.java:1714) at org.hibernate.query.internal.AbstractProducedQuery.executeUpdate(AbstractProducedQuery.java:1696) at org.springframework.data.jpa.repository.query.JpaQueryExecution$ModifyingExecution.doExecute(JpaQueryExecution.java:241) at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:90) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:156) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:144) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:137) at org.springframework.data.repository.core.support.RepositoryMethodInvoker.invoke(RepositoryMethodInvoker.java:121) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:160) at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:139) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.data.projection.DefaultMethodInvokingMethodInterceptor.invoke(DefaultMethodInvokingMethodInterceptor.java:81) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137) ... 200 common frames omitted Caused by: java.sql.SQLTransientConnectionException: (conn=134538047) Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:128) at com.oceanbase.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:202) at com.oceanbase.jdbc.OceanBaseStatement.executeExceptionEpilogue(OceanBaseStatement.java:320) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:257) at com.oceanbase.jdbc.JDBC4PreparedStatement.execute(JDBC4PreparedStatement.java:171) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeUpdate(JDBC4PreparedStatement.java:205) at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61) at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java) at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.executeUpdate(ResultSetReturnImpl.java:197) ... 224 common frames omitted Caused by: com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.util.exceptions.OceanBaseSqlException.of(OceanBaseSqlException.java:79) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:205) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.exceptionWithQuery(AbstractQueryProtocol.java:186) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:401) at com.oceanbase.jdbc.JDBC4PreparedStatement.executeInternal(JDBC4PreparedStatement.java:248) ... 229 common frames omitted Caused by: java.sql.SQLException: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readErrorPacket(AbstractQueryProtocol.java:2366) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.readPacket(AbstractQueryProtocol.java:2229) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:2117) at com.oceanbase.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:399) ... 230 common frames omitted 2025-10-27 11:39:39.791 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.c.cache.helper.BinlogQueryCache : Finish init binlog cache. 2025-10-27 11:39:39.791 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: obProxyQueryCache 2025-10-27 11:39:39.791 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.c.cache.helper.ObProxyQueryCache : Init obproxy cache. 2025-10-27 11:39:39.803 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.c.cache.helper.ObProxyQueryCache : Finished init obproxy cache. 2025-10-27 11:39:39.803 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.s.common.scheduled.CacheSchedules : Refresh cache bean: hostQueryCache 2025-10-27 11:39:39.803 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.core.cache.helper.HostQueryCache : Init host cache. 2025-10-27 11:39:39.812 INFO 2458 --- [ocp-schedules-19,fe1bbfbcf532736d,2b5c93be9395a933] c.o.o.core.cache.helper.HostQueryCache : Finished init host cache. 2025-10-27 11:39:40.007 INFO 2458 --- [ocp-schedules-11,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:39:40.008 INFO 2458 --- [ocp-schedules-1,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:39:40.056 INFO 2458 --- [Check obproxy1,4b2ead3703edf3a8,69ff70100d55a7d6] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:39:40.056 INFO 2458 --- [Check obproxy1,4b2ead3703edf3a8,69ff70100d55a7d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:39:40.056 INFO 2458 --- [Collect obproxy1,9f0e12f8d4386db0,60d62c7d1eff525d] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:39:40.056 INFO 2458 --- [Collect obproxy1,9f0e12f8d4386db0,60d62c7d1eff525d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:39:40.106 INFO 2458 --- [Check obproxy1,4b2ead3703edf3a8,69ff70100d55a7d6] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:39:40.107 INFO 2458 --- [Collect obproxy1,9f0e12f8d4386db0,60d62c7d1eff525d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:39:40.109 INFO 2458 --- [Check obproxy1,4b2ead3703edf3a8,69ff70100d55a7d6] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:39:40.109 INFO 2458 --- [Check obproxy1,4b2ead3703edf3a8,69ff70100d55a7d6] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:39:40.110 INFO 2458 --- [Collect obproxy1,9f0e12f8d4386db0,60d62c7d1eff525d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:39:40.113 INFO 2458 --- [Collect obproxy1,9f0e12f8d4386db0,60d62c7d1eff525d] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:39:40.694 WARN 2458 --- [ocp-async-5,d51ac800bedcbce3,ddae559aa857ca34] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:40.694 INFO 2458 --- [ocp-async-1,84709ec62a2ec102,26bdae17f902dd4d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:40.720 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:40.720 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:39:40.720 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:39:40.720 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:39:40.720 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:39:40.720 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:39:40.730 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:40.732 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:40.734 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:40.736 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:40.736 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:40.738 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:40.740 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:40.741 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:40.743 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:40.744 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:40.795 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:39:40.798 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:39:40.800 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:39:40.809 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:39:40.811 INFO 2458 --- [ForkJoinPool.commonPool-worker-2,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:39:40.814 INFO 2458 --- [ocp-schedules-3,cba1f9b04c813057,20b392e18a43415d] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:39:41.311 INFO 2458 --- [ocp-schedules-29,79378df9c04315f1,5b4d68266d499faf] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:39:41.370 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:39:41.370 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:39:41.381 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,07067b1a1e5180d8,d3d61e2349306e85] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=07067b1a1e5180d8, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:39:41.382 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-7,07067b1a1e5180d8,d3d61e2349306e85] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=07067b1a1e5180d8, duration=1 ms] 2025-10-27 11:39:41.427 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=root 2025-10-27 11:39:41.484 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:41.485 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:39:41.499 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:41.500 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:39:41.537 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=root 2025-10-27 11:39:41.560 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:41.561 INFO 2458 --- [ocp-schedules-18,28bd00e66f790028,aaba9c7102cdb79f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:39:41.707 INFO 2458 --- [prometheus-minute-schedule-1,ab5a6aa6ef2a6b70,ada955671cecc532] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:41.714 INFO 2458 --- [prometheus-minute-schedule-1,ab5a6aa6ef2a6b70,ada955671cecc532] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:41.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:41.714108+08:00, lastProcessTime=2025-10-27T11:39:31.716351+08:00 2025-10-27 11:39:41.941 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:41.714108+08:00 2025-10-27 11:39:42.694 WARN 2458 --- [ocp-async-8,a7e4c4dca3c30b0a,e774027480583f8f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:42.694 INFO 2458 --- [ocp-async-10,7788c55659923469,ac5fe7924871b051] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:42.731 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:42.732 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:44.355 INFO 2458 --- [ocp-schedules-29,10da173e1e7887d4,e0d505d5a071d2f1] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:39:44.694 WARN 2458 --- [ocp-async-7,6a3310edd44f49d9,ebbd40e59857c4f8] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:44.694 INFO 2458 --- [ocp-async-4,84e7a995924b4a10,d913362440e03bd3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:44.707 INFO 2458 --- [prometheus-minute-schedule-1,e50d58d6fd40d9fe,8dc9c632099a768a] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:44.714 INFO 2458 --- [prometheus-minute-schedule-1,e50d58d6fd40d9fe,8dc9c632099a768a] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:45.102 INFO 2458 --- [prometheus-second-schedule-1,3a8407c58a8726ad,8569a1712b165a6f] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:39:45.732 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:45.733 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:46.298 INFO 2458 --- [ocp-schedules-3,b59bfcebfe652442,e7670c237809756a] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:46.393 WARN 2458 --- [scanner-112226495-1,b59bfcebfe652442,c3bd2d7256d29a80] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:46.496 INFO 2458 --- [alarm-detect-20,b59bfcebfe652442,93de741c4b17ecb5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536355 ~ 1761536376 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536370, value=1.828)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.828, values={1761536370=1.828}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:46.694 WARN 2458 --- [ocp-async-6,f2c9489ba6ea7a01,77c85e098d493a66] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:46.694 INFO 2458 --- [ocp-async-6,aaf0ce15cc744184,8c62d040a6fc2990] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:46.793 INFO 2458 --- [alarm-detect-20,b59bfcebfe652442,5ed9ec43c300412e] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536355 ~ 1761536376 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536370, value=84.401)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536370, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536370=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:47.234 INFO 2458 --- [ocp-schedules-26,64d56d14f0c238a0,ca54277d9d3c8641] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant mini compaction 2025-10-27 11:39:47.234 INFO 2458 --- [ocp-schedules-26,64d56d14f0c238a0,ca54277d9d3c8641] t.b.t.CollectAllTenantMiniCompactionTask : Begin to collect all tenants mini compaction info 2025-10-27 11:39:47.300 INFO 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:39:47.300 INFO 2458 --- [pool-883-thread-2,64d56d14f0c238a0,5ac20c5458eeb6e4] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect mini compaction info 2025-10-27 11:39:47.375 INFO 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:47.376 INFO 2458 --- [pool-883-thread-2,64d56d14f0c238a0,5ac20c5458eeb6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:47.377 INFO 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761536087000000, 1761536387000000] 2025-10-27 11:39:47.379 INFO 2458 --- [pool-883-thread-2,64d56d14f0c238a0,5ac20c5458eeb6e4] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, MIN(unix_timestamp(`start_time`)*1000000) AS min_start_time, MAX(unix_timestamp(`finish_time`)*1000000) AS max_finish_time, SUM(occupy_size) AS occupy_size, SUM(total_row_count) AS total_row_count, COUNT(1) AS tablet_count FROM GV$OB_TABLET_COMPACTION_HISTORY WHERE type = 'MINI_MERGE' AND unix_timestamp(`finish_time`)*1000000 > ? AND unix_timestamp(`finish_time`)*1000000 <= ? GROUP BY tenant_id;, args: [1761536087000000, 1761536387000000] 2025-10-27 11:39:47.422 WARN 2458 --- [pool-883-thread-2,64d56d14f0c238a0,5ac20c5458eeb6e4] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase2, ob tenant id:1001 2025-10-27 11:39:47.439 INFO 2458 --- [pool-883-thread-2,64d56d14f0c238a0,5ac20c5458eeb6e4] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:39:47.487 WARN 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1001 2025-10-27 11:39:47.487 WARN 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1003 2025-10-27 11:39:47.487 WARN 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] o.i.t.TenantCompactionCollectServiceImpl : Tenant not found in ocp, cluster name:myoceanbase, ob tenant id:1005 2025-10-27 11:39:47.496 INFO 2458 --- [pool-883-thread-1,64d56d14f0c238a0,d99d6993d5e7080f] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect mini compaction info 2025-10-27 11:39:47.497 INFO 2458 --- [ocp-schedules-26,64d56d14f0c238a0,ca54277d9d3c8641] t.b.t.CollectAllTenantMiniCompactionTask : Finish to collect all tenants mini compaction info 2025-10-27 11:39:47.707 INFO 2458 --- [prometheus-minute-schedule-1,7516abc884aa37d3,1bce1ca8de0d7666] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:47.715 INFO 2458 --- [prometheus-minute-schedule-1,7516abc884aa37d3,1bce1ca8de0d7666] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:47.841 INFO 2458 --- [ocp-schedules-1,4928156d4ed801d1,d7b315205fe05b98] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:47.975 INFO 2458 --- [ocp-schedules-15,ed5744462caeeff6,d3abdd6e02e46441] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:39:47.984 INFO 2458 --- [ocp-schedules-15,ed5744462caeeff6,d3abdd6e02e46441] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:47.984 INFO 2458 --- [ocp-schedules-15,ed5744462caeeff6,d3abdd6e02e46441] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:39:48.546 INFO 2458 --- [ocp-async-5,4d7522f6dd01e60e,40b51c9b746fd6a9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:48.695 WARN 2458 --- [ocp-async-8,44a7ed7b934d1d7b,6f72f3962fe3d1cd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:48.695 INFO 2458 --- [ocp-async-2,3a3de34430c60b3d,81e3802d3a3e006c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:48.733 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:48.734 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:49.748 INFO 2458 --- [ocp-schedules-23,0d4da03630b5b84f,31e63255e2ce64bd] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:49.752 INFO 2458 --- [ocp-schedules-23,0d4da03630b5b84f,31e63255e2ce64bd] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:39:50.386 INFO 2458 --- [ocp-schedules-18,32c52b38137076cb,bdf1a9a30fba05de] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:39:50.386+08:00, clusters=[1, 2] 2025-10-27 11:39:50.401 INFO 2458 --- [ocp-schedules-18,32c52b38137076cb,bdf1a9a30fba05de] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:39:50.403 INFO 2458 --- [ocp-schedules-18,32c52b38137076cb,bdf1a9a30fba05de] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:39:50.695 INFO 2458 --- [ocp-async-7,f3b804c4607338ca,c899d6855c807056] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:50.695 WARN 2458 --- [ocp-async-9,6790db13126089f8,a41ed4dc4b92d91f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:50.707 INFO 2458 --- [prometheus-minute-schedule-1,b763750734228a40,cb78911ace86608d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:50.713 INFO 2458 --- [prometheus-minute-schedule-1,b763750734228a40,cb78911ace86608d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:51.409 INFO 2458 --- [ocp-schedules-4,20a42ddaedef86c3,18359f742214c4d9] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:51.409 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [ocp-schedules-9,b93ff3f34f04fe02,4ce8e29b19e3a766] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:39:51.410 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:39:51.411 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:39:51.423 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:51.424 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:51.424 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:51.425 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:51.425 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:51.426 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:51.426 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:51.428 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:51.428 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:51.429 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:51.429 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:51.429 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:51.431 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:51.431 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:51.431 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:51.431 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:51.431 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:51.433 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:51.434 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:51.435 INFO 2458 --- [ocp-schedules-27,5286a66ca86543e6,0a0ec9a25721878a] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:39:51.436 INFO 2458 --- [host-check-clockdiff1,5286a66ca86543e6,5f893b2d04318334] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:39:51.436 INFO 2458 --- [host-check-clockdiff2,5286a66ca86543e6,a461971866c4730c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:39:51.436 INFO 2458 --- [host-check-clockdiff3,5286a66ca86543e6,042ca4483ed76c4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:39:51.436 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:51.437 INFO 2458 --- [host-check-clockdiff5,5286a66ca86543e6,636554e1ec54e9a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:39:51.437 INFO 2458 --- [host-check-clockdiff4,5286a66ca86543e6,3bc0ca1d6a9e8f6b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:39:51.442 INFO 2458 --- [ocp-schedules-28,d693268e82ba39e2,98d561365d4da3c1] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:39:51.460 INFO 2458 --- [host-check-clockdiff3,5286a66ca86543e6,042ca4483ed76c4a] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536391 0 0 , error= 2025-10-27 11:39:51.459 INFO 2458 --- [host-check-clockdiff5,5286a66ca86543e6,636554e1ec54e9a9] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536391 0 0 , error= 2025-10-27 11:39:51.464 INFO 2458 --- [host-check-clockdiff2,5286a66ca86543e6,a461971866c4730c] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536391 -1 -1 , error= 2025-10-27 11:39:51.464 INFO 2458 --- [host-check-clockdiff4,5286a66ca86543e6,3bc0ca1d6a9e8f6b] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536391 0 0 , error= 2025-10-27 11:39:51.464 INFO 2458 --- [host-check-clockdiff1,5286a66ca86543e6,5f893b2d04318334] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536391 0 0 , error= 2025-10-27 11:39:51.499 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:39:51.501 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:39:51.501073535+08:00 2025-10-27 11:39:51.502 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:39:51.501+08:00 2025-10-27 11:39:51.503 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:39:51.503 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:39:51.504 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:39:51.505 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:39:51.505162227+08:00 2025-10-27 11:39:51.506 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:39:51.505+08:00 2025-10-27 11:39:51.507 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:39:51.507 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-agent-status5,20a42ddaedef86c3,6a928bd6feedb1f5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-agent-status1,20a42ddaedef86c3,4b231c63fc4b82e5] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:39:51.508 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:39:51.509 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:39:51.509389025+08:00 2025-10-27 11:39:51.509 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:39:51.509+08:00 2025-10-27 11:39:51.510 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:39:51.511 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:39:51.511 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:39:51.511 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:39:51.511 INFO 2458 --- [host-check-agent-status2,20a42ddaedef86c3,83e92991f1d814a4] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status4,20a42ddaedef86c3,29c8ace6ad906482] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:39:51.513 INFO 2458 --- [host-check-agent-status3,20a42ddaedef86c3,02ed5933fb621dd1] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:39:51.520 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:39:51.501+08:00 and status AVAILABLE 2025-10-27 11:39:51.522 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:39:51.522 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:39:51.522583060+08:00 2025-10-27 11:39:51.523 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:39:51.522+08:00 2025-10-27 11:39:51.525 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:39:51.526 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:39:51.509+08:00 and status AVAILABLE 2025-10-27 11:39:51.528 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:39:51.505+08:00 and status AVAILABLE 2025-10-27 11:39:51.531 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:39:51.532 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:39:51.522+08:00 and status AVAILABLE 2025-10-27 11:39:51.533 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:39:51.533 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:39:51.534 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:39:51.533386402+08:00 2025-10-27 11:39:51.535 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:39:51.534+08:00 2025-10-27 11:39:51.537 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:39:51.545 INFO 2458 --- [ocp-schedules-21,6b26069fc28f3f3e,9cc9e58f653aaa29] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:39:51.546 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:39:51.534+08:00 and status AVAILABLE 2025-10-27 11:39:51.548 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:39:51.549 INFO 2458 --- [ocp-schedules-19,5b910e46f1076cfe,f2cfa5dcba019778] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:39:51.550 INFO 2458 --- [ocp-schedules-21,6b26069fc28f3f3e,9cc9e58f653aaa29] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:51.551 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:39:51.551 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:39:51.553 INFO 2458 --- [ocp-schedules-19,5b910e46f1076cfe,f2cfa5dcba019778] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:51.554 INFO 2458 --- [ocp-schedules-18,6a4cae7da0498d6f,2d4907f2344ada98] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:39:51.554 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:39:51.555 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:51.555 INFO 2458 --- [ocp-schedules-19,5b910e46f1076cfe,f2cfa5dcba019778] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:39:51.557 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:39:51.558 INFO 2458 --- [ocp-schedules-18,6a4cae7da0498d6f,2d4907f2344ada98] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:51.560 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:51.562 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:51.565 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:51.565 INFO 2458 --- [ocp-schedules-7,95ee85cddf6141fd,7e45672851548a92] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:39:51.568 INFO 2458 --- [ocp-schedules-25,040401c8db23055b,761c019b2c043cfb] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:39:51.569 INFO 2458 --- [ocp-schedules-15,4e298e69e9dd2f07,80776b4aad3cd505] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:39:51.577 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:39:51.578 INFO 2458 --- [ocp-schedules-25,040401c8db23055b,761c019b2c043cfb] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:39:51.580 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:39:51.581 INFO 2458 --- [ocp-schedules-15,4e298e69e9dd2f07,80776b4aad3cd505] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:39:51.599 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.600 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:51.601 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:39:51.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.631 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:51.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.633 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:51.634 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:51.634 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:51.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:39:51.636 INFO 2458 --- [ocp-schedules-21,6b26069fc28f3f3e,9cc9e58f653aaa29] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:39:51.639 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:51.646 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:51.649 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:39:51.653 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:39:51.662 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:39:51.665 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:39:51.670 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:39:51.688 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:39:51.688 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:51.689 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:39:51.689 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.690 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:39:51.692 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:39:51.692 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:39:51.696 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.697 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:51.702 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:39:51.716 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:39:51.714923+08:00, lastProcessTime=2025-10-27T11:39:41.714108+08:00 2025-10-27 11:39:51.732 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:51.734 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:51.735 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:51.736 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:51.739 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:51.747 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:39:51.749 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:51.749 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:39:51.752 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:39:51.760 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:39:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:51.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:51.766 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.774 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:39:51.775 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:51.780 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:39:51.781 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:39:51.782 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:39:51.788 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:39:51.793 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:51.794 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.794 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:39:51.799 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:39:51.802 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:51.803 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:51.807 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:39:51.814 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:39:51.815 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:51.826 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:51.831 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:51.833 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:51.835 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:39:51.840 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:39:51.846 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:39:51.852 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:39:51.853 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:51.869 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.870 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:51.872 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:39:51.874 INFO 2458 --- [host-check-host-status4,b93ff3f34f04fe02,532707325a00dbbe] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:39:51.877 INFO 2458 --- [host-check-host-status5,b93ff3f34f04fe02,576d3d3dbab9a9b0] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:39:51.884 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:39:51.887 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:39:51.890 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:39:51.896 INFO 2458 --- [host-check-host-status3,b93ff3f34f04fe02,15802a0f6c903d87] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:39:51.902 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:39:51.905 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:39:51.909 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:39:51.909 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:51.910 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.911 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:39:51.912 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:51.915 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:39:51.916 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:51.921 INFO 2458 --- [host-check-host-status1,b93ff3f34f04fe02,13dbb7893c6954b2] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:39:51.940 INFO 2458 --- [host-check-host-status2,b93ff3f34f04fe02,08748c1b528cde30] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:39:51.949 INFO 2458 --- [ocp-schedules-5,a384ebaf40f039ee,d1ce0a020e8aac64] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:39:51.958 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:51.958 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:51.964 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:39:51.981 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:39:51.714923+08:00 2025-10-27 11:39:51.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:51.995 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:52.005 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:39:52.006 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.006 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:39:52.008 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.009 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:39:52.024 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:39:52.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:52.029 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:52.051 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:52.052 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.053 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:39:52.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:52.067 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:39:52.076 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:39:52.076 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:39:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:52.101 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:39:52.146 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:39:52.185 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:52.186 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.186 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:39:52.190 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:39:52.199 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:39:52.210 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:39:52.213 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:39:52.272 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:39:52.336 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:39:52.336 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.337 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:39:52.338 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.339 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:39:52.377 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:39:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:39:52.415 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:39:52.416 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:39:52.455 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:39:52.455 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535802505000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:39:52.456 INFO 2458 --- [ocp-schedules-18,6a4cae7da0498d6f,2d4907f2344ada98] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:39:52.695 WARN 2458 --- [ocp-async-5,b4bf92289ee34477,4bf9ba0d298e9378] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:52.695 INFO 2458 --- [ocp-async-6,e0c9d184bae62982,5ce9026598a6eeb8] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:39:53.237 INFO 2458 --- [ocp-schedules-15,5e112f9789bec06e,5a51a9a1958c7474] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20324,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5081,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:39:53.238 INFO 2458 --- [ocp-schedules-15,5e112f9789bec06e,5a51a9a1958c7474] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:39:53.707 INFO 2458 --- [prometheus-minute-schedule-1,91b881fca885f3a4,39b32212f94c3e3d] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:53.714 INFO 2458 --- [prometheus-minute-schedule-1,91b881fca885f3a4,39b32212f94c3e3d] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:54.695 WARN 2458 --- [ocp-async-10,073437fb7a1c8584,dc6c028eb5f46500] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:54.695 INFO 2458 --- [ocp-async-9,97c2693fa0270e38,4cdfa872daf965a6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:54.735 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:54.736 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:55.948 INFO 2458 --- [ocp-schedules-30,f72c011c87bd8e22,3c532171962ec03b] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:39:56.695 WARN 2458 --- [ocp-async-7,720c2878d3c1652b,d34f78c8b1b874a6] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:56.695 INFO 2458 --- [ocp-async-3,65a919ffbd366411,370c86ceeaa244fc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:56.707 INFO 2458 --- [prometheus-minute-schedule-1,dc3b1277889fe475,4a1182d088b8b976] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:56.714 INFO 2458 --- [prometheus-minute-schedule-1,dc3b1277889fe475,4a1182d088b8b976] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:56.801 INFO 2458 --- [ocp-schedules-25,41ba9f0ce63154c0,debadcdfeaf72a69] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:39:56.830 WARN 2458 --- [scanner-112226685-1,41ba9f0ce63154c0,15bf271407e7df33] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:39:56.869 INFO 2458 --- [alarm-detect-5,41ba9f0ce63154c0,b378025f8e52d2d8] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536326 ~ 1761536386 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536340, value=204674.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204674.56, values={1761536340=204674.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:39:56.870 INFO 2458 --- [alarm-detect-5,41ba9f0ce63154c0,b378025f8e52d2d8] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_compaction, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:39:56.968 INFO 2458 --- [alarm-detect-12,41ba9f0ce63154c0,7da10368d5cd1cfc] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536326 ~ 1761536386 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536340, value=204675.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204675.912, values={1761536340=204675.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:39:56.969 INFO 2458 --- [alarm-detect-12,41ba9f0ce63154c0,7da10368d5cd1cfc] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_no_frozen, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:39:56.983 INFO 2458 --- [alarm-detect-13,41ba9f0ce63154c0,862c9ba92dd35a20] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536326 ~ 1761536386 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536340, value=93.272)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.272, values={1761536340=93.272}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:39:56.985 INFO 2458 --- [alarm-detect-13,41ba9f0ce63154c0,862c9ba92dd35a20] c.o.o.a.core.detect.AlarmRuleDetector : detectByMetric done, alarmType=ob_tenant_log_disk_usage_high, needSendingCount=1, detectAlarmsCount=1, matchedTargets=[alarm_template_id=0:ob_cluster=myoceanbase-1760510187:tenant_name=ocp_monitor-2] 2025-10-27 11:39:57.001 INFO 2458 --- [alarm-detect-9,41ba9f0ce63154c0,52f62bf462b67cb5] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536370 ~ 1761536386 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536385, value=78.872)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536385, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536385=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:39:57.736 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:39:57.737 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:39:57.848 INFO 2458 --- [ocp-schedules-30,0ad9ecac8881211f,0612018c7fc30fe7] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:39:58.228 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,9b44390427345b40,488de4ce01676daa] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=9b44390427345b40, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:39:58.228 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-1,9b44390427345b40,488de4ce01676daa] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=9b44390427345b40, duration=1 ms] 2025-10-27 11:39:58.546 INFO 2458 --- [ocp-async-4,2b7822821d2db3b1,d121ea11a105df81] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:39:58.565 INFO 2458 --- [ocp-schedules-13,6544e7ef32ea736e,dbbc8a853cc85270] c.o.o.s.c.scheduled.ClusterSchedules : Sync root server list. 2025-10-27 11:39:58.570 INFO 2458 --- [ocp-schedules-13,6544e7ef32ea736e,dbbc8a853cc85270] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:39:58.613 INFO 2458 --- [ocp-schedules-13,6544e7ef32ea736e,dbbc8a853cc85270] .o.o.s.t.b.c.AbstractShardingClusterTask : Refresh root service list executed completed, allCount=2, successCount=2 2025-10-27 11:39:58.695 INFO 2458 --- [ocp-async-5,2686be42906f33df,630cc18ec6c8fb75] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:39:58.695 WARN 2458 --- [ocp-async-6,4b3503693746df9e,4c337cbc1f94864c] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:39:59.634 INFO 2458 --- [ocp-schedules-4,d06b9b5429e72b4a,9e10d04e55e4ca50] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:39:59.639 INFO 2458 --- [ocp-schedules-4,d06b9b5429e72b4a,9e10d04e55e4ca50] c.o.o.a.service.OcpAlarmSearchService : alarmTargetStat [] 2025-10-27 11:39:59.641 INFO 2458 --- [ocp-schedules-11,2f2e77b8ab00d91c,9d4bb75951a0b364] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:39:59.707 INFO 2458 --- [prometheus-minute-schedule-1,d6fa4005621bc744,3dbab5eb3815b985] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536400, interval=60 2025-10-27 11:39:59.715 INFO 2458 --- [prometheus-minute-schedule-1,d6fa4005621bc744,3dbab5eb3815b985] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:39:59.760 INFO 2458 --- [ocp-schedules-17,7ac4d55e4ae9f995,4c762cbbbb5fa30d] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:39:59.764 INFO 2458 --- [ocp-schedules-17,7ac4d55e4ae9f995,4c762cbbbb5fa30d] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:00.337 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:35:01.0, nextRun=2025-10-27 11:40:00.0, currentDate=Mon Oct 27 11:40:00 CST 2025 2025-10-27 11:40:00.337 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999977, name=DiagFuncUpdateSchedule, templateName=DiagFuncUpdateSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:35:01.0, nextRunTime=2025-10-27 11:40:00.0} 2025-10-27 11:40:00.370 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=DiagFuncUpdateSchedule, nodeCount=1, taskCount=1 2025-10-27 11:40:00.370 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='DiagFuncUpdateSchedule', taskDefinitionId=999977, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:40:00.370+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Update diag function, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.diagnosis.task.DiagFuncUpdateTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:40:00.421 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:35:01.0, nextRun=2025-10-27 11:40:00.0, currentDate=Mon Oct 27 11:40:00 CST 2025 2025-10-27 11:40:00.421 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999985, name=SqlParseSchedule, templateName=SqlParseSchedule, scheduleRule=0 */5 * * * *, lastRunTime=2025-10-27 11:35:01.0, nextRunTime=2025-10-27 11:40:00.0} 2025-10-27 11:40:00.511 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.p.s.internal.SqlParseTaskSchedule : get Clusters for parse:[1, 2] 2025-10-27 11:40:00.565 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.p.s.internal.SqlParseTaskSchedule : HighRiskSql Parse is ready 2025-10-27 11:40:00.571 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=SqlParseTaskSchedule, nodeCount=1, taskCount=1 2025-10-27 11:40:00.571 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,2b55d935267b13dc] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='SqlParseTaskSchedule', taskDefinitionId=999985, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={cluster_ids=[[1,2]]}}, createTime=null, startTime=2025-10-27T11:40:00.571+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Sql Parse Task, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.perf.sql.internal.task.SqlParseTask, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:40:00.696 INFO 2458 --- [ocp-async-2,ea649257d0e050d0,10eee86f2ba7752b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:00.696 WARN 2458 --- [ocp-async-9,acc83461ee5a3123,622905cd7ddb3c3f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:00.737 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:00.738 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:00.803 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,17cd6b3f3265d1dc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9181, operation:EXECUTE, state: RUNNING 2025-10-27 11:40:00.820 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,17cd6b3f3265d1dc] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9182, operation:EXECUTE, state: RUNNING 2025-10-27 11:40:00.851 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:40:00.852 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9181, context=Context{parallelIdx=-1, stringMap={task_instance_id=9113, task_operation=execute, latest_execution_start_time=2025-10-27T11:40:00.790+08:00, sub_task_instance_name=Update diag function, sub_task_instance_id=9181}, listMap={}}, executor=10.40.65.146 2025-10-27 11:40:00.856 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:40:00.861 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9182, context=Context{parallelIdx=0, stringMap={subtask_splitter=cluster_ids, task_instance_id=9114, task_operation=execute, latest_execution_start_time=2025-10-27T11:40:00.817+08:00, sub_task_instance_name=Sql Parse Task, sub_task_instance_id=9182}, listMap={cluster_ids=[[1,2]]}}, executor=10.40.65.146 2025-10-27 11:40:00.861 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:1 2025-10-27 11:40:00.956 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.058 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1002,1761523200000000,2000,30000000 2025-10-27 11:40:01.106 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.191 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.299 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.468 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.621 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=2, name=ocp_meta, obTenantId=1002, status=NORMAL} 2025-10-27 11:40:01.627 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:01.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:01.716205+08:00, lastProcessTime=2025-10-27T11:39:51.714923+08:00 2025-10-27 11:40:01.806 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1004,1761523200000000,2000,30000000 2025-10-27 11:40:01.822 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=3, name=ocp_monitor, obTenantId=1004, status=NORMAL} 2025-10-27 11:40:02.021 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:02.039 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510187,myoceanbase,1006,1761523200000000,2000,30000000 2025-10-27 11:40:02.080 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:01.716205+08:00 2025-10-27 11:40:02.113 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase 2025-10-27 11:40:02.136 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase 2025-10-27 11:40:02.137 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.ocp.diagnosis.model.NeuralNetwork : W:[12188.005973021298], b:[-2.1447346366359664E13] 2025-10-27 11:40:02.144 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.166 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.186 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.201 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=4, name=test, obTenantId=1006, status=NORMAL} 2025-10-27 11:40:02.201 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.p.sql.internal.task.SqlParseTask : parse sql for cluster:2 2025-10-27 11:40:02.214 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.287 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.340 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.341 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : querySqlTextWithParams:1760510188,myoceanbase2,1002,1761523200000000,2000,30000000 2025-10-27 11:40:02.345 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.perf.sql.internal.SqlParseService : parse 0 sqls for tenant:ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL} 2025-10-27 11:40:02.376 INFO 2458 --- [subtask-executor28,8964d58b78e8b300,fe8828985fdb829e] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9182, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:40:02.470 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.503 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,89b9c54a51f6dec6,a84026a7fe923437] c.o.o.s.c.trace.RequestTracingAspect : API: [POST /services?Action=ObRootServiceInfo&User_ID=alibaba&UID=ocpmaster&ObRegion=myoceanbase&version=2&ObClusterId=1760510187, client=10.40.65.146, traceId=89b9c54a51f6dec6, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.updateRootServiceInfo(String,Long,String,Long,Integer,String), args=myoceanbase,1760510187,2,{"ObRegion":"myoceanbase","ObCluster":"myoceanbase","ObRegionId":1760510187,"ObClusterId":1760510187,"Type":"PRIMARY","timestamp":1761536402501314,"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"ReadonlyRsList":[]},] 2025-10-27 11:40:02.510 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.DiagnosisFuncUpdateService : update diag func fo cluster:myoceanbase2 2025-10-27 11:40:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,89b9c54a51f6dec6,a84026a7fe923437] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info ClusterCacheInfo(clusterName=myoceanbase, obClusterId=1760510187, rootServerJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761536402504000}, groupName=myoceanbase, obproxyClusters=null, tenantInfos=[]) into config server cache 2025-10-27 11:40:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,89b9c54a51f6dec6,a84026a7fe923437] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Put cluster info into config server cache success 2025-10-27 11:40:02.514 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,89b9c54a51f6dec6,a84026a7fe923437] c.o.ocp.configurl.ConfigUrlService : updateRootServer, clusterName=myoceanbase, clusterId=1760510187, affectRows=1 2025-10-27 11:40:02.516 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-3,89b9c54a51f6dec6,a84026a7fe923437] c.o.o.s.c.trace.RequestTracingAspect : API OK: [POST /services client=10.40.65.146, traceId=89b9c54a51f6dec6, duration=14 ms] 2025-10-27 11:40:02.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.d.o.DiskUsageTimeRemainingOperator : do least Square for myoceanbase2 2025-10-27 11:40:02.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.ocp.diagnosis.model.NeuralNetwork : W:[448.47108155422495], b:[-7.893195228861522E11] 2025-10-27 11:40:02.560 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,f43ef989b639a78b] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9181, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:40:02.696 INFO 2458 --- [ocp-async-4,e53cfacd605d9690,1082b7c9692714e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:02.696 WARN 2458 --- [ocp-async-3,0801287231b05eec,03fb7ad5508e752b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:02.707 INFO 2458 --- [prometheus-minute-schedule-1,acb22d64810b6de4,0f74f1cb5a4ac620] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:02.734 INFO 2458 --- [prometheus-minute-schedule-1,acb22d64810b6de4,0f74f1cb5a4ac620] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:02.951 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9113, status:SUCCESS 2025-10-27 11:40:02.964 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9113, status:SUCCESS 2025-10-27 11:40:02.973 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9114, status:SUCCESS 2025-10-27 11:40:02.976 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9114, status:SUCCESS 2025-10-27 11:40:03.738 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:03.741 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:04.696 INFO 2458 --- [ocp-async-8,282fb63cc0ff209f,e6448e3eeacf9138] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:04.696 WARN 2458 --- [ocp-async-5,2fc49c03195de8b2,b6bbc2b8f6759503] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:04.760 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,f4d4b0d576eb6926] c.o.o.c.t.engine.util.SpringScheduler : Schedule time arrived, lastRun=2025-10-27 11:30:04.0, nextRun=2025-10-27 11:40:04.0, currentDate=Mon Oct 27 11:40:04 CST 2025 2025-10-27 11:40:04.760 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,f4d4b0d576eb6926] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Schedule ready task, entity=TaskDefinitionEntity{id=999999, name=Partitioning daemon, templateName=PartitioningDaemonSchedule, scheduleRule=4 */10 * * * *, lastRunTime=2025-10-27 11:30:04.0, nextRunTime=2025-10-27 11:40:04.0} 2025-10-27 11:40:04.784 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,f4d4b0d576eb6926] c.o.o.c.t.e.c.worker.SubtaskGenerator : generate tasks, templateName=Partition rollover daemon, nodeCount=1, taskCount=1 2025-10-27 11:40:04.784 INFO 2458 --- [task-coordinator3,8964d58b78e8b300,f4d4b0d576eb6926] c.o.o.c.t.e.c.worker.ReadyTaskWorker : Save task instance, entity=TaskInstanceEntity{id=null, name='Partition rollover daemon', taskDefinitionId=999999, state=RUNNING, type=SYS_SCHEDULED, clusterId=null, tenantId=null, hostId=null, creator='system', executor='10.40.65.146', argument=Argument{parallelIdx=-1, stringMap={prohibit_rollback=false}, listMap={}}, createTime=null, startTime=2025-10-27T11:40:04.784+08:00, endTime=null, subtasks=[SubtaskInstanceEntity{id=null, name=Partition rollover daemon, state=READY, operation=EXECUTE, className=com.oceanbase.ocp.service.partitioning.PartitioningDaemonSchedule$Task, seriesId=0, startTime=null, endTime=null}], template=null, traceId='8964d58b78e8b300', operation=EXECUTE} 2025-10-27 11:40:04.862 INFO 2458 --- [subtask-coordinator2,8964d58b78e8b300,da7b287282e30445] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9183, operation:EXECUTE, state: RUNNING 2025-10-27 11:40:04.908 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.service.iam.user.UserService : user 200 login with organization 10000000 2025-10-27 11:40:04.910 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.c.t.e.runner.JavaSubtaskRunner : Run subtask, id=9183, context=Context{parallelIdx=-1, stringMap={task_instance_id=9115, task_operation=execute, latest_execution_start_time=2025-10-27T11:40:04.860+08:00, sub_task_instance_name=Partition rollover daemon, sub_task_instance_id=9183}, listMap={}}, executor=10.40.65.146 2025-10-27 11:40:04.924 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.c.d.lock.jdbc.DefaultLockManager : Acquired distributed lock for key part-worker 2025-10-27 11:40:04.924 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Metadb 2025-10-27 11:40:04.925 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.926 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:40:04.934 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.934 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:04.936 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.937 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table profile_credential_sharing 2025-10-27 11:40:04.982 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.983 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:04.984 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.985 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_cache_info 2025-10-27 11:40:04.992 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Metadb 2025-10-27 11:40:04.992 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.s.p.PartitionRolloverService : [OCP167006] Start watching scope, scope=Monitordb 2025-10-27 11:40:04.993 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.994 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:40:04.996 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.996 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:04.997 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:04.998 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_0 2025-10-27 11:40:05.023 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.024 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.025 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.025 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_0 2025-10-27 11:40:05.046 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.047 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.048 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.049 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_1 2025-10-27 11:40:05.066 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.067 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.068 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.068 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_1 2025-10-27 11:40:05.085 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.086 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.087 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.088 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat_2 2025-10-27 11:40:05.106 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.107 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.108 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.109 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_stat_2 2025-10-27 11:40:05.129 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.130 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.131 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.131 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_stat 2025-10-27 11:40:05.152 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.153 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.154 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.154 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sqltext 2025-10-27 11:40:05.163 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.164 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.165 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.165 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sqltext 2025-10-27 11:40:05.174 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.175 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.176 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.176 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_dynamic_sql 2025-10-27 11:40:05.183 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.183 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.184 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.185 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_high_risk_sql 2025-10-27 11:40:05.192 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.193 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.193 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.194 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_stat_0 2025-10-27 11:40:05.202 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.203 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.204 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.205 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_stat_0 2025-10-27 11:40:05.214 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.215 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.216 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.216 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan 2025-10-27 11:40:05.223 INFO 2458 --- [ocp-schedules-16,3f80178dbd26bed8,9958f5e6b0f8ba7a] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:40:05.224 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.225 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.226 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.226 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan 2025-10-27 11:40:05.234 INFO 2458 --- [ocp-schedules-16,3f80178dbd26bed8,9958f5e6b0f8ba7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:05.234 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.235 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.236 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.236 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_plan_cache_plan_explain 2025-10-27 11:40:05.244 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.244 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.245 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.246 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_plan_cache_plan_explain 2025-10-27 11:40:05.254 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.255 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.256 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.256 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample 2025-10-27 11:40:05.270 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.270 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.271 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.272 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_v2_sql_audit_sample 2025-10-27 11:40:05.286 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.286 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.287 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.288 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_audit_sample_statistics 2025-10-27 11:40:05.304 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.305 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:40:05.307 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.308 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.309 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.310 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis_task 2025-10-27 11:40:05.317 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.318 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.319 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.319 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_perf_sql_diagnosis 2025-10-27 11:40:05.328 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.328 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.329 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.330 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_active_session_stat_0 2025-10-27 11:40:05.337 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.337 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.337 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:40:05.337 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.338 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.339 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_snapshot 2025-10-27 11:40:05.345 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.346 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.347 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.347 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cluster 2025-10-27 11:40:05.354 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.354 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.355 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.355 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_tenant 2025-10-27 11:40:05.359 INFO 2458 --- [ocp-schedules-16,3f80178dbd26bed8,9958f5e6b0f8ba7a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:40:05.362 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.362 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.363 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.364 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_server 2025-10-27 11:40:05.371 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.371 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.372 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.373 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_replica 2025-10-27 11:40:05.380 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.381 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.382 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.383 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sysstat 2025-10-27 11:40:05.389 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.390 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.391 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.391 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_row_lock 2025-10-27 11:40:05.398 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.398 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.399 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.400 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_latch 2025-10-27 11:40:05.407 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.407 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.408 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.409 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_cache 2025-10-27 11:40:05.415 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.416 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.417 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_parameter 2025-10-27 11:40:05.424 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.424 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.425 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.426 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_rpc 2025-10-27 11:40:05.432 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.433 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.433 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.434 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_packet 2025-10-27 11:40:05.441 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.441 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.442 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.443 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_clog 2025-10-27 11:40:05.449 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.450 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.451 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_location_cache 2025-10-27 11:40:05.457 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.458 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.459 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.459 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_gts 2025-10-27 11:40:05.465 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.466 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.466 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.467 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_system_event 2025-10-27 11:40:05.473 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.474 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.475 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.475 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_memory 2025-10-27 11:40:05.482 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.483 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.484 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.484 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_hit 2025-10-27 11:40:05.490 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.491 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.491 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.492 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_sql_plan_monitor 2025-10-27 11:40:05.502 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.502 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.503 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.504 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_trans_stat_v2 2025-10-27 11:40:05.512 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.513 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.513 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.514 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_system_event 2025-10-27 11:40:05.520 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.521 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.522 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.522 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_cluster_server_event 2025-10-27 11:40:05.529 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.529 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.530 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.531 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_1 2025-10-27 11:40:05.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.537 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.538 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.539 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ocp_metric_data_60 2025-10-27 11:40:05.544 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.544 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.545 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.545 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_hour_data 2025-10-27 11:40:05.550 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.551 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.551 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.552 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_daily_data 2025-10-27 11:40:05.557 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.557 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.558 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.559 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_second 2025-10-27 11:40:05.564 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.564 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.565 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.566 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_minute 2025-10-27 11:40:05.571 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.572 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.572 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.573 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_hour 2025-10-27 11:40:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.579 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.580 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.581 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_daily 2025-10-27 11:40:05.586 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.587 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.587 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.588 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table metric_data_async 2025-10-27 11:40:05.594 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.594 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.595 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.596 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table sql_aggregate_data 2025-10-27 11:40:05.600 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.600 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.601 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.602 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_hist_partition_stats_compressed 2025-10-27 11:40:05.607 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.608 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:05.609 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:05.609 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: show create table ob_tenant_mini_compaction_info 2025-10-27 11:40:05.616 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.s.p.PartitionRolloverService : [OCP167007] Finish watching scope, scope=Monitordb 2025-10-27 11:40:05.633 INFO 2458 --- [subtask-executor30,8964d58b78e8b300,568e01cc19b17b05] c.o.o.c.t.e.c.w.subtask.SubtaskAccessor : Set state for subtask: 9183, operation:EXECUTE, state: SUCCESSFUL 2025-10-27 11:40:05.707 INFO 2458 --- [prometheus-minute-schedule-1,4b9a9fec72447279,92b8359becc08394] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:05.718 INFO 2458 --- [prometheus-minute-schedule-1,4b9a9fec72447279,92b8359becc08394] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:05.999 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:06.000 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:40:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:06.002 INFO 2458 --- [sql-rollup-daemon,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:40:06.004 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule start 2025-10-27 11:40:06.028 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : begin update audit event, taskId:9115, status:SUCCESS 2025-10-27 11:40:06.032 INFO 2458 --- [task-coordinator1,,] c.o.ocp.service.audit.AuditEventService : audit event not found, taskId:9115, status:SUCCESS 2025-10-27 11:40:06.068 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=1, name=myoceanbase, obClusterId=1760510187, status=RUNNING} 2025-10-27 11:40:06.111 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : got cluster:ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING} 2025-10-27 11:40:06.136 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-3) 2025-10-27 11:40:06.136 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-2) 2025-10-27 11:40:06.136 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase2:1760510188-1) 2025-10-27 11:40:06.137 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Submit concurrent group: (myoceanbase:1760510187-1) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-0,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=3, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=0->1, c=0) - not enough cached interval (source=2025-10-15T18:04:30, target=2025-10-15T18:03:30, granule=120000ms, delay=60000ms) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-2,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=2, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:40:06.141 INFO 2458 --- [sql-rollup-worker-3,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase:1760510187, server=1, level=1->2, c=0) - not enough cached interval (source=2025-10-15T18:03:30, target=2025-10-15T18:23:17, granule=600000ms, delay=-1187000ms) 2025-10-27 11:40:06.142 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:06.143 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW VARIABLES LIKE 'version_comment' 2025-10-27 11:40:06.145 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:06.145 INFO 2458 --- [sql-rollup-worker-1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ now() 2025-10-27 11:40:06.218 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=0) - source=2025-10-27T11:37:30, start=2025-10-27T11:35:00, end=2025-10-27T11:37:30, updates=245, granule=120000ms, delay=150000ms 2025-10-27 11:40:06.226 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=0->1, c=1) - not enough cached interval (source=2025-10-27T11:37:30, target=2025-10-27T11:37:30, granule=120000ms, delay=0ms) 2025-10-27 11:40:06.298 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Rollup server-side (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=0) - source=2025-10-27T11:37:30, start=2025-10-27T11:22:30, end=2025-10-27T11:32:30, updates=387, granule=600000ms, delay=900000ms 2025-10-27 11:40:06.696 WARN 2458 --- [ocp-async-10,9d33ec2ca389287e,f2afaed46c6f9235] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:06.696 INFO 2458 --- [ocp-async-9,c457c5af0f233ac6,0db6eebc902bf8e5] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:06.738 INFO 2458 --- [sql-rollup-worker-1,,] o.o.p.s.i.c.SqlAuditRawStatRollupService : Try rollup subsequent (cluster=myoceanbase2:1760510188, server=1, level=1->2, c=1) - not enough cached interval (source=2025-10-27T11:37:30, target=2025-10-27T11:32:30, granule=600000ms, delay=300000ms) 2025-10-27 11:40:06.739 INFO 2458 --- [sql-rollup-daemon,,] s.i.c.SqlAuditRawStatRollupDaemonService : Rollup schedule end, elapsed=735ms, period=120000ms 2025-10-27 11:40:06.741 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:06.742 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:07.024 INFO 2458 --- [ocp-schedules-9,fa5f29cc918deedf,a4ea256d00910cc2] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:07.059 WARN 2458 --- [scanner-112227142-1,fa5f29cc918deedf,445f965c73426cdf] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:07.086 INFO 2458 --- [alarm-detect-3,fa5f29cc918deedf,af30d8d4a151a4aa] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536370 ~ 1761536397 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536385, value=1.571)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.571, values={1761536385=1.571}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:07.854 INFO 2458 --- [ocp-schedules-5,199f0b635bb01f42,5ade5dbc66190d39] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:07.990 INFO 2458 --- [ocp-schedules-24,b9257db171bf787d,3c86b32856263b22] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:40:08.000 INFO 2458 --- [ocp-schedules-24,b9257db171bf787d,3c86b32856263b22] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:08.000 INFO 2458 --- [ocp-schedules-24,b9257db171bf787d,3c86b32856263b22] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:08.546 INFO 2458 --- [ocp-async-2,69d7155fd452882b,8229a354b56973f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:08.696 INFO 2458 --- [ocp-async-3,65043d6ce5d3ba63,c7c58701543dbd99] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:08.696 WARN 2458 --- [ocp-async-7,5dbb9ca17aea3aa8,1e4c911d0a3953fd] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:08.707 INFO 2458 --- [prometheus-minute-schedule-1,ac0159019b3d07ac,06f36209d2460848] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:08.715 INFO 2458 --- [prometheus-minute-schedule-1,ac0159019b3d07ac,06f36209d2460848] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:09.742 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:09.744 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:09.772 INFO 2458 --- [ocp-schedules-25,220de5eb525d09b1,26c594537801056e] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:09.776 INFO 2458 --- [ocp-schedules-25,220de5eb525d09b1,26c594537801056e] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:10.018 INFO 2458 --- [ocp-schedules-9,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:40:10.018 INFO 2458 --- [ocp-schedules-27,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:40:10.075 INFO 2458 --- [Collect obproxy1,b59b52309cda12c7,7a51f2ebc84b98c2] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:40:10.075 INFO 2458 --- [Check obproxy1,35da883257f0f417,e0ba5bdc27e6f2fa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:40:10.075 INFO 2458 --- [Collect obproxy1,b59b52309cda12c7,7a51f2ebc84b98c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:40:10.075 INFO 2458 --- [Check obproxy1,35da883257f0f417,e0ba5bdc27e6f2fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:40:10.110 INFO 2458 --- [Collect obproxy1,b59b52309cda12c7,7a51f2ebc84b98c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:40:10.111 INFO 2458 --- [Check obproxy1,35da883257f0f417,e0ba5bdc27e6f2fa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:40:10.114 INFO 2458 --- [Collect obproxy1,b59b52309cda12c7,7a51f2ebc84b98c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:40:10.115 INFO 2458 --- [Check obproxy1,35da883257f0f417,e0ba5bdc27e6f2fa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:40:10.115 INFO 2458 --- [Check obproxy1,35da883257f0f417,e0ba5bdc27e6f2fa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:40:10.117 INFO 2458 --- [Collect obproxy1,b59b52309cda12c7,7a51f2ebc84b98c2] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:40:10.184 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:40:10.696 INFO 2458 --- [ocp-async-3,838da6a8ae765937,81fcb3a8d982ee53] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:10.696 WARN 2458 --- [ocp-async-5,ceb37088c6070159,4c93115ecd3b713b] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:11.707 INFO 2458 --- [prometheus-minute-schedule-1,9a26fc2b57aa79dc,77954a26ca75a10e] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:11.714 INFO 2458 --- [prometheus-minute-schedule-1,9a26fc2b57aa79dc,77954a26ca75a10e] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:11.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:11.714094+08:00, lastProcessTime=2025-10-27T11:40:01.716205+08:00 2025-10-27 11:40:11.737 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=1000, affectRows=1000 2025-10-27 11:40:11.741 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=994, validCount=992 2025-10-27 11:40:11.816 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=1000, affectRows=1000 2025-10-27 11:40:11.820 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=999, validCount=995 2025-10-27 11:40:11.833 ERROR 2458 --- [metric-archive-second-4,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134570103) Duplicate entry '835370603008958464-1761536280' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134570103) Duplicate entry '835370603008958464-1761536280' for key 'PRIMARY' 2025-10-27 11:40:11.935 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:11.714094+08:00 2025-10-27 11:40:12.562 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536340, nodeCount=817, affectRows=817 2025-10-27 11:40:12.696 INFO 2458 --- [ocp-async-8,8c14c6762421533f,663e9f8c8d78a664] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:12.696 WARN 2458 --- [ocp-async-10,0b70100e03c2974a,1ffaa774042004e0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:12.744 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:12.745 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:13.069 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=1000, affectRows=1000 2025-10-27 11:40:13.074 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=992, validCount=991 2025-10-27 11:40:13.154 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=1000, affectRows=1000 2025-10-27 11:40:13.158 INFO 2458 --- [metric-archive-second-8,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=1000, validCount=993 2025-10-27 11:40:13.161 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=81, affectRows=81 2025-10-27 11:40:13.161 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=80, validCount=80 2025-10-27 11:40:14.697 INFO 2458 --- [ocp-async-7,b05d34f12771c60b,8efaaf38c60ede23] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:14.697 WARN 2458 --- [ocp-async-4,ffc66b0bca232f43,eeb7d7b6eb900c71] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:14.707 INFO 2458 --- [prometheus-minute-schedule-1,6ab1d49d8670b07e,3b9df1c227da8331] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:14.717 INFO 2458 --- [prometheus-minute-schedule-1,6ab1d49d8670b07e,3b9df1c227da8331] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:15.745 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:15.746 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:15.948 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536340, nodeCount=1000, affectRows=1000 2025-10-27 11:40:15.952 INFO 2458 --- [metric-archive-second-4,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536340, nodeCount=1000, validCount=987 2025-10-27 11:40:16.697 INFO 2458 --- [ocp-async-7,1f14a2c3a65a7c0f,e4309e85c88a0c7b] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:16.697 WARN 2458 --- [ocp-async-5,0b93fa40c5e420b4,7dcc5611d3461897] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:16.725 INFO 2458 --- [ocp-schedules-22,514276ce11ea4d9a,30103e5fe7a177d0] c.o.o.s.common.scheduled.PerfSchedules : Collect trans stat for alarm, time=2025-10-27T11:40:16.725+08:00, clusters=[1, 2] 2025-10-27 11:40:16.729 INFO 2458 --- [ocp-schedules-22,514276ce11ea4d9a,30103e5fe7a177d0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase, trans stat result is empty. 2025-10-27 11:40:16.729 INFO 2458 --- [ocp-schedules-22,514276ce11ea4d9a,30103e5fe7a177d0] c.o.o.p.t.i.alarm.TransStatDataSource : query trans stat of obCluster myoceanbase2, trans stat result is empty. 2025-10-27 11:40:17.201 INFO 2458 --- [ocp-schedules-9,9f6b9adba1a7e41e,f1d41d6bf0d77f1d] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:17.267 WARN 2458 --- [scanner-112228843-1,9f6b9adba1a7e41e,44096afda206b2c3] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:17.369 INFO 2458 --- [alarm-detect-20,9f6b9adba1a7e41e,603994d9ebc38770] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536385 ~ 1761536407 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536400, value=1.618)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.618, values={1761536400=1.618}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:17.439 INFO 2458 --- [alarm-detect-19,9f6b9adba1a7e41e,6a5ea8640085e592] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536385 ~ 1761536407 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536400, value=60.944)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536400, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536400=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:17.707 INFO 2458 --- [prometheus-minute-schedule-1,34497bdcaec0df5a,fabd6a01556eeaa5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:17.715 INFO 2458 --- [prometheus-minute-schedule-1,34497bdcaec0df5a,fabd6a01556eeaa5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:17.860 INFO 2458 --- [ocp-schedules-24,15b470ac48f60058,537c556fe75347d1] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:18.546 INFO 2458 --- [ocp-async-5,22b36b0bbd98c8cb,208bbc6b48ce57d3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:18.602 INFO 2458 --- [ocp-schedules-11,212bada6e0f444a3,118dd037deac8c24] c.o.o.s.c.scheduled.TenantSchedules : Sync all tenant info schedule. 2025-10-27 11:40:18.609 INFO 2458 --- [ocp-schedules-11,212bada6e0f444a3,118dd037deac8c24] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:18.688 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:40:18.688 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.691 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:40:18.697 INFO 2458 --- [ocp-async-10,85189e9a4e659b71,ee65b3080e4c2742] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:18.697 WARN 2458 --- [ocp-async-2,4148dc7867eec391,6458ffb4618a8496] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:18.700 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.700 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.727 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.728 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.746 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:18.747 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:18.749 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.HintQueryer : add hints for sql, hints=[query_timeout(60000000)], after=SELECT /*+ query_timeout(60000000) */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:40:18.749 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.753 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.754 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ query_timeout(60000000) ocp_query */ TENANT_ID, TENANT_NAME, TENANT_TYPE, PRIMARY_ZONE, LOCALITY, COMPATIBILITY_MODE, STATUS, IN_RECYCLEBIN, (CASE WHEN LOCKED = 'YES' THEN 1 ELSE 0 END) AS LOCKED, TIMESTAMPDIFF(SECOND, CREATE_TIME, now()) AS exist_seconds , ARBITRATION_SERVICE_STATUS, REPLACE(`SWITCHOVER_STATUS`, ' ', '_') as SWITCHOVER_STATUS, LOG_MODE, SYNC_SCN, RECOVERY_UNTIL_SCN, TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_TYPE IN ('SYS', 'USER') 2025-10-27 11:40:18.754 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.756 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.757 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.782 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.782 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.785 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.785 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.809 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.810 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.818 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.818 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.846 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.846 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1] 2025-10-27 11:40:18.853 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.853 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.854 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.854 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:18.855 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.855 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:18.856 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.857 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:18.858 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.859 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:18.859 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.860 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:18.861 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.862 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:18.862 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=2 2025-10-27 11:40:18.865 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:40:18.886 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.886 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.912 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.913 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1002] 2025-10-27 11:40:18.914 INFO 2458 --- [Sync all tenant information2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:18.915 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.915 INFO 2458 --- [Sync all tenant information2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:40:18.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.917 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:40:18.925 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=5, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:18.925 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=5, userWhitelist=% 2025-10-27 11:40:18.925 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=6, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:18.925 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=6, userWhitelist=% 2025-10-27 11:40:18.927 INFO 2458 --- [Sync all tenant information2,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=2 2025-10-27 11:40:18.927 INFO 2458 --- [Sync all tenant information2,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:40:18.946 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.947 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:40:18.970 INFO 2458 --- [ocp-schedules-9,8bbac2d19482a81d,84f9b4ba4fdcdab7] c.o.o.s.c.scheduled.TenantSchedules : Collect all tenant major compaction 2025-10-27 11:40:18.970 INFO 2458 --- [ocp-schedules-9,8bbac2d19482a81d,84f9b4ba4fdcdab7] o.s.t.b.t.CollectAllTenantCompactionTask : Begin to collect all tenants major compaction info 2025-10-27 11:40:18.976 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.976 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1004] 2025-10-27 11:40:18.979 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:18.979 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:40:19.007 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.008 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:40:19.019 INFO 2458 --- [Collect all tenants major compactions1,8bbac2d19482a81d,3df38764f35ff871] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:40:19.020 INFO 2458 --- [Collect all tenants major compactions2,8bbac2d19482a81d,4a36c21ef572e6f6] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:40:19.020 INFO 2458 --- [Collect all tenants major compactions3,8bbac2d19482a81d,8d2ce8430ce27345] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:40:19.020 INFO 2458 --- [Collect all tenants major compactions5,8bbac2d19482a81d,388f34d0fe7a98d3] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:40:19.020 INFO 2458 --- [Collect all tenants major compactions6,8bbac2d19482a81d,9cf7542c3e3dc9d2] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:40:19.021 INFO 2458 --- [Collect all tenants major compactions4,8bbac2d19482a81d,55d3514cc2538e1d] o.i.t.TenantCompactionCollectServiceImpl : Begin to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:40:19.036 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.036 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, SERVICE_NAME, SERVICE_STATUS FROM CDB_OB_SERVICES WHERE TENANT_ID = ?, args: [1006] 2025-10-27 11:40:19.051 INFO 2458 --- [Collect all tenants major compactions5,8bbac2d19482a81d,388f34d0fe7a98d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.053 INFO 2458 --- [Collect all tenants major compactions5,8bbac2d19482a81d,388f34d0fe7a98d3] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:40:19.053 INFO 2458 --- [Collect all tenants major compactions6,8bbac2d19482a81d,9cf7542c3e3dc9d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.055 INFO 2458 --- [Collect all tenants major compactions6,8bbac2d19482a81d,9cf7542c3e3dc9d2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:40:19.058 INFO 2458 --- [Collect all tenants major compactions6,8bbac2d19482a81d,9cf7542c3e3dc9d2] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:sys 2025-10-27 11:40:19.060 INFO 2458 --- [Collect all tenants major compactions5,8bbac2d19482a81d,388f34d0fe7a98d3] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase2, tenant:test2 2025-10-27 11:40:19.071 INFO 2458 --- [Collect all tenants major compactions1,8bbac2d19482a81d,3df38764f35ff871] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.072 INFO 2458 --- [Collect all tenants major compactions3,8bbac2d19482a81d,8d2ce8430ce27345] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.074 INFO 2458 --- [Collect all tenants major compactions1,8bbac2d19482a81d,3df38764f35ff871] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1006] 2025-10-27 11:40:19.073 INFO 2458 --- [Collect all tenants major compactions3,8bbac2d19482a81d,8d2ce8430ce27345] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1002] 2025-10-27 11:40:19.074 INFO 2458 --- [Collect all tenants major compactions2,8bbac2d19482a81d,4a36c21ef572e6f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.078 INFO 2458 --- [Collect all tenants major compactions4,8bbac2d19482a81d,55d3514cc2538e1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.079 INFO 2458 --- [Collect all tenants major compactions4,8bbac2d19482a81d,55d3514cc2538e1d] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1] 2025-10-27 11:40:19.080 INFO 2458 --- [Collect all tenants major compactions2,8bbac2d19482a81d,4a36c21ef572e6f6] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION WHERE tenant_id = ?, args: [1004] 2025-10-27 11:40:19.081 INFO 2458 --- [Collect all tenants major compactions1,8bbac2d19482a81d,3df38764f35ff871] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:test 2025-10-27 11:40:19.082 INFO 2458 --- [Collect all tenants major compactions3,8bbac2d19482a81d,8d2ce8430ce27345] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_meta 2025-10-27 11:40:19.085 INFO 2458 --- [Collect all tenants major compactions4,8bbac2d19482a81d,55d3514cc2538e1d] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:sys 2025-10-27 11:40:19.086 INFO 2458 --- [Collect all tenants major compactions2,8bbac2d19482a81d,4a36c21ef572e6f6] o.i.t.TenantCompactionCollectServiceImpl : Finish to collect major compaction info for cluster:myoceanbase, tenant:ocp_monitor 2025-10-27 11:40:19.087 INFO 2458 --- [ocp-schedules-9,8bbac2d19482a81d,84f9b4ba4fdcdab7] o.s.t.b.t.CollectAllTenantCompactionTask : Finish to collect all tenants major compaction info 2025-10-27 11:40:19.090 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.091 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:19.095 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.096 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:19.096 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.097 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:19.098 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.099 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:19.099 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.100 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:19.101 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.101 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1002, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:19.102 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.103 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:19.103 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.104 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:19.105 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1004, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:19.106 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.107 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_TENANT_LOCALITY] 2025-10-27 11:40:19.108 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, SHRINK_RESOURCE_POOL_UNIT_NUM] 2025-10-27 11:40:19.109 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.110 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, JOB_ID, JOB_TYPE, JOB_STATUS, PROGRESS FROM oceanbase.DBA_OB_TENANT_JOBS WHERE TENANT_ID = ? AND JOB_TYPE = ? ORDER BY JOB_ID DESC LIMIT 1, args: [1006, ALTER_RESOURCE_TENANT_UNIT_NUM] 2025-10-27 11:40:19.110 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648001] sync all tenant whitelist start, clusterId=1 2025-10-27 11:40:19.113 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648111] default whitelist refreshed, defaultWhitelist=10.40.65.146 2025-10-27 11:40:19.193 INFO 2458 --- [Sync all tenant information1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:19.193 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:19.194 INFO 2458 --- [Sync all tenant information1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ TENANT_ID, NAME, VALUE FROM oceanbase.CDB_OB_SYS_VARIABLES WHERE name = ?, args: [ob_tcp_invited_nodes] 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=1, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=1, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=2, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=2, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=3, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=3, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648201] user whitelist calculated, tenantId=4, realWhitelist=%, defaultWhitelist=127.0.0.1,10.40.65.146, userWhitelist=% 2025-10-27 11:40:19.225 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648202] user whitelist updated, tenantId=4, userWhitelist=% 2025-10-27 11:40:19.229 INFO 2458 --- [Sync all tenant information1,,] c.o.o.o.i.t.TenantWhitelistServiceImpl : [OCP648002] sync all tenant whitelist done, clusterId=1 2025-10-27 11:40:19.229 INFO 2458 --- [Sync all tenant information1,,] .o.s.t.b.t.SyncAllTenantInfoShardingTask : [OCP606] sync tenants success, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:40:19.229 INFO 2458 --- [ocp-schedules-11,212bada6e0f444a3,118dd037deac8c24] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all tenant information executed completed, allCount=2, successCount=2 2025-10-27 11:40:19.784 INFO 2458 --- [ocp-schedules-21,a5545d408c5e596f,9bcb3d503b956482] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:19.788 INFO 2458 --- [ocp-schedules-21,a5545d408c5e596f,9bcb3d503b956482] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:20.697 INFO 2458 --- [ocp-async-4,3d859728ca4b1eaa,2f85f1d513ab2c93] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:20.697 WARN 2458 --- [ocp-async-9,ca3ff4272cf4c3c5,21823eb77ec0f643] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:20.707 INFO 2458 --- [prometheus-minute-schedule-1,cbaf4e462d22c169,842a050dcfd5849c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:20.714 INFO 2458 --- [prometheus-minute-schedule-1,cbaf4e462d22c169,842a050dcfd5849c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:21.047 INFO 2458 --- [ocp-schedules-28,559fd5ab2b61c43e,49acf9d5e8770120] c.o.o.s.common.scheduled.PerfSchedules : Oas sql curing, clusters=[] 2025-10-27 11:40:21.434 INFO 2458 --- [ocp-schedules-27,f9914dda279f12f9,5815404771d9e737] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:21.434 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:40:21.434 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:40:21.434 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:40:21.434 INFO 2458 --- [ocp-schedules-13,23c6ec7b5357370e,49a7288539b656ef] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:21.434 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:40:21.434 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:40:21.435 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:40:21.435 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:40:21.435 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:40:21.435 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:40:21.435 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:40:21.442 INFO 2458 --- [ocp-schedules-8,937a50b18ce58704,ff53b2b8678a1917] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:40:21.454 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:21.454 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:21.454 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:21.454 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:21.455 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:21.455 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:21.455 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:21.458 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:21.458 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:21.459 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:21.459 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:21.459 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:21.460 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:21.460 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:21.460 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:21.461 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:21.463 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:21.464 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:21.466 INFO 2458 --- [ocp-schedules-14,1367cc50f1acfba1,1bb9235888fc00b3] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:21.468 INFO 2458 --- [host-check-clockdiff1,1367cc50f1acfba1,ebfe05d8c69715f5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:40:21.468 INFO 2458 --- [host-check-clockdiff2,1367cc50f1acfba1,c6cd932c0d586e81] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:40:21.468 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:21.481 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:21.487 INFO 2458 --- [host-check-clockdiff3,1367cc50f1acfba1,c5b4861e8df15bc8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:40:21.488 INFO 2458 --- [host-check-clockdiff4,1367cc50f1acfba1,e8eed755b5ce5311] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:40:21.503 INFO 2458 --- [host-check-clockdiff1,1367cc50f1acfba1,ebfe05d8c69715f5] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536421 0 0 , error= 2025-10-27 11:40:21.517 INFO 2458 --- [host-check-clockdiff5,1367cc50f1acfba1,604499210e19bead] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:40:21.536 INFO 2458 --- [host-check-clockdiff2,1367cc50f1acfba1,c6cd932c0d586e81] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536421 -1 -1 , error= 2025-10-27 11:40:21.538 INFO 2458 --- [host-check-clockdiff3,1367cc50f1acfba1,c5b4861e8df15bc8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536421 0 0 , error= 2025-10-27 11:40:21.545 INFO 2458 --- [ocp-schedules-24,35e8d28d8ba04c53,1fc17fd85d38f0fc] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:40:21.549 INFO 2458 --- [ocp-schedules-22,ec247222b89f6a1b,fca3349382d84ed1] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:40:21.552 INFO 2458 --- [host-check-clockdiff4,1367cc50f1acfba1,e8eed755b5ce5311] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536421 -1 -1 , error= 2025-10-27 11:40:21.552 INFO 2458 --- [host-check-clockdiff5,1367cc50f1acfba1,604499210e19bead] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536421 1 1 , error= 2025-10-27 11:40:21.554 INFO 2458 --- [ocp-schedules-6,510eab2e5fb723d7,9e94bae30a706908] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:40:21.563 INFO 2458 --- [ocp-schedules-6,510eab2e5fb723d7,9e94bae30a706908] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:21.565 INFO 2458 --- [ocp-schedules-22,ec247222b89f6a1b,fca3349382d84ed1] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:21.565 INFO 2458 --- [ocp-schedules-22,ec247222b89f6a1b,fca3349382d84ed1] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:40:21.566 INFO 2458 --- [ocp-schedules-24,35e8d28d8ba04c53,1fc17fd85d38f0fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:21.567 INFO 2458 --- [ocp-schedules-5,5211a1fbc4fffd66,3f294cb598cc7860] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:40:21.573 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:40:21.575 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:40:21.575840779+08:00 2025-10-27 11:40:21.578 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:40:21.578 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:40:21.575+08:00 2025-10-27 11:40:21.579 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:40:21.578026056+08:00 2025-10-27 11:40:21.579 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:40:21.580 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:40:21.580 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:40:21.579+08:00 2025-10-27 11:40:21.580 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:40:21.581 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:40:21.582 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:40:21.582 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:40:21.582281868+08:00 2025-10-27 11:40:21.582 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:40:21.582494700+08:00 2025-10-27 11:40:21.583 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:40:21.583 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:40:21.582+08:00 2025-10-27 11:40:21.585 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:40:21.582+08:00 2025-10-27 11:40:21.585 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:40:21.585 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:40:21.585 INFO 2458 --- [host-check-agent-status4,f9914dda279f12f9,0da805ac0a6e68de] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:40:21.586 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:40:21.586 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:40:21.586 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:40:21.586 INFO 2458 --- [host-check-agent-status1,f9914dda279f12f9,9325a76c051a95ea] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-agent-status3,f9914dda279f12f9,fb6cd91fdef50837] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:40:21.579+08:00 and status AVAILABLE 2025-10-27 11:40:21.590 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:40:21.591 INFO 2458 --- [host-check-agent-status2,f9914dda279f12f9,dde6421f57efc926] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:40:21.591 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:40:21.575+08:00 and status AVAILABLE 2025-10-27 11:40:21.591 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:40:21.591 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:40:21.591273266+08:00 2025-10-27 11:40:21.592 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:40:21.592+08:00 2025-10-27 11:40:21.593 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:40:21.582+08:00 and status AVAILABLE 2025-10-27 11:40:21.593 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:40:21.582+08:00 and status AVAILABLE 2025-10-27 11:40:21.593 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:40:21.593 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:40:21.593 INFO 2458 --- [host-check-agent-status5,f9914dda279f12f9,5e78cc0336ed7533] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:40:21.595 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:40:21.597 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:40:21.599 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:40:21.599 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:40:21.600 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:40:21.592+08:00 and status AVAILABLE 2025-10-27 11:40:21.605 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:40:21.613 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.614 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:21.615 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:40:21.616 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:21.621 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:21.621 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:21.621 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:21.623 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:21.624 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:21.624 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:21.626 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:21.627 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:21.629 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.630 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:21.630 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:21.636 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:40:21.636 INFO 2458 --- [ocp-schedules-24,35e8d28d8ba04c53,1fc17fd85d38f0fc] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:40:21.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.638 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:21.682 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:21.683 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.683 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:21.684 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:21.686 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:21.687 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:21.695 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:21.695 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:21.696 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.697 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:40:21.698 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:40:21.699 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:40:21.710 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:40:21.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:21.713511+08:00, lastProcessTime=2025-10-27T11:40:11.714094+08:00 2025-10-27 11:40:21.716 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:40:21.721 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:40:21.737 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:40:21.745 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:40:21.747 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:40:21.747 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:21.748 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:40:21.749 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:21.758 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:40:21.762 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:21.763 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.764 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:21.765 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.768 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:40:21.769 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:21.770 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:21.773 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:21.775 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:21.777 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:21.785 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:40:21.785 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.786 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:21.786 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:40:21.787 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:21.788 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:21.790 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:40:21.797 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:40:21.799 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:40:21.818 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:40:21.820 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:21.823 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:40:21.824 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:40:21.836 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:40:21.840 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:40:21.846 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:21.848 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:40:21.858 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:21.859 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:40:21.860 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:21.860 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:40:21.866 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:21.870 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:21.870 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:21.872 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:40:21.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:21.876 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:40:21.881 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.882 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:21.883 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:21.887 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:40:21.887 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:40:21.889 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:40:21.892 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:21.892 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:21.914 INFO 2458 --- [host-check-host-status4,23c6ec7b5357370e,295e026a4809b10a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:40:21.921 INFO 2458 --- [host-check-host-status5,23c6ec7b5357370e,4a488907cbe2fe63] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:40:21.921 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:40:21.928 INFO 2458 --- [host-check-host-status3,23c6ec7b5357370e,80d9800390a2c0e1] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:40:21.931 INFO 2458 --- [host-check-host-status2,23c6ec7b5357370e,423d1cfbdcde9bca] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:40:21.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:21.948 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:21.951 INFO 2458 --- [host-check-host-status1,23c6ec7b5357370e,9ed77c053fef8d5e] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:40:21.954 INFO 2458 --- [ocp-schedules-11,1a16cb5e4b315a4a,95341a74f6a29d36] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:40:21.957 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:40:21.990 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:21.713511+08:00 2025-10-27 11:40:21.999 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:40:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.000 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:40:22.002 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:22.002 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:22.002 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.003 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:40:22.019 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:40:22.038 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:22.038 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:22.042 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:22.042 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.043 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:40:22.065 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:40:22.065 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:40:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:22.073 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:22.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:22.105 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:22.148 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:22.195 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:22.196 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.196 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:22.200 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:40:22.211 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:40:22.219 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:22.222 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:22.271 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:40:22.347 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:40:22.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.348 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:40:22.385 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.386 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:40:22.425 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:40:22.463 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:22.463 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:22.464 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:40:22.504 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:40:22.504 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761536402504000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:40:22.505 INFO 2458 --- [ocp-schedules-6,510eab2e5fb723d7,9e94bae30a706908] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:40:22.697 INFO 2458 --- [ocp-async-6,1e040dde785c78da,fc64fde35c0247f1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:22.697 WARN 2458 --- [ocp-async-5,02218b33f0b7c875,dff85015fa47c92a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:23.707 INFO 2458 --- [prometheus-minute-schedule-1,1555a48ad17be02d,06c6463045de5cdb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:23.714 INFO 2458 --- [prometheus-minute-schedule-1,1555a48ad17be02d,06c6463045de5cdb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:24.697 INFO 2458 --- [ocp-async-3,8a11ac1017aad163,2127a2ba2f0b504d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:24.697 WARN 2458 --- [ocp-async-8,977691c03fdb407d,d51fddeb3491669a] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:24.749 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:24.750 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:26.201 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : Starting PLAN_CHANGE_DETECTOR, Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195} 2025-10-27 11:40:26.211 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR : for [Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null 2025-10-27 11:40:26.211 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:26.214 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:26.214 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=2, name=ocp_meta, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:40:26.214 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:26.218 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:26.219 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:26.219 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=1, name=sys, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:40:26.219 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:26.221 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:26.234 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 44 diagnose targets Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:26.315 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=5, name=sys, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T14:41:29+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:40:26.315 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:26.505 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761450026505000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=67E38E3073F798EBB9224740FEBB44D1, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:40:26.505Z, endTime=2025-10-27T03:38:26.195Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:40:26.515 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761450026515000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=5, obTenantId=1, tenantName=sys, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:40:26.515Z, endTime=2025-10-27T03:38:26.195Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=5, name=sys, obTenantId=1, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:40:26.537 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 0 2025-10-27 11:40:26.537 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:40:26.556 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 263 2025-10-27 11:40:26.556 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:40:26.562 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:26.562 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:26.572 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:40:26.572 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:40:26.579 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:26.579 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:26.586 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:40:26.586 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:40:26.595 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:26.595 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=67E38E3073F798EBB9224740FEBB44D1, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:26.599 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records,use plan union hash 2025-10-27 11:40:26.599 INFO 2458 --- [plan_change_detector8,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 0 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:40:26.505, end=2025-10-27T11:38:26.195}, clusterId=2, tenantId=5, sqlId=67E38E3073F798EBB9224740FEBB44D1, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:40:26.607 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:26.607 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1, obServerId=null, obDbId=201001, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:26.611 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:40:26.697 INFO 2458 --- [ocp-async-3,1e93d249c37be3ae,f22bd0a484754092] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:26.697 WARN 2458 --- [ocp-async-9,a60b0cf0c60e15e8,24f2ee45b15c4784] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:26.707 INFO 2458 --- [prometheus-minute-schedule-1,b236e26d7e4825b6,e5a39da167304c81] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:26.713 INFO 2458 --- [prometheus-minute-schedule-1,b236e26d7e4825b6,e5a39da167304c81] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:27.007 INFO 2458 --- [plan_change_detector3,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:40:26.515, end=2025-10-27T11:38:26.195}, clusterId=2, tenantId=5, sqlId=F4812CAE86BAF2CE7F72DDBB80AB1404, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:40:27.008 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:27.011 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:27.011 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=4, name=test, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-16T11:11:22+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1006, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:40:27.011 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:27.015 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:27.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:27.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=3, name=ocp_monitor, clusterName=myoceanbase, obClusterId=1760510187, clusterId=1, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-15T17:58:49+08:00, primaryZone=RANDOM, zoneList=zone1;zone2;zone3, locality=FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1004, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=null, options=null), right=null) 2025-10-27 11:40:27.016 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:27.019 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql plan 2025-10-27 11:40:27.023 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : PERFORMANCE_DEGRADATION_AFTER_PLAN_CHANGED found out 0 diagnose targets Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null), Interval{start=2025-10-27T11:36:25.36, end=2025-10-27T11:38:26.195},PlanChangePerfDegradationDetectService.Config(enabled=true, sysTenantEnabled=true, level=0, compareMin=5, cpuTimeMultiply={0=50.0, 1=20.0, 10=10.0, 100=3.0, 1000=2.0}, limit=2000, cpuTimeLimitMs=1.0, alarm=true, autoFlushPlanCache=false, minElapsedTimeMsForFlushingPlanCache=50.0) 2025-10-27 11:40:27.023 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : Found out 0 dubious sqls of Pair(left=Tenant(id=6, name=test2, clusterName=myoceanbase2, obClusterId=1760510188, clusterId=2, arbitrationServiceId=null, clusterType=PRIMARY, clusterStatus=RUNNING, syncStatus=VALID, mode=MYSQL, createTime=2025-10-17T15:42:53+08:00, primaryZone=RANDOM, zoneList=zone1, locality=FULL{1}@zone1, status=NORMAL, operatingTaskInstanceId=null, statusAnalysis=null, locked=false, readonly=false, obVersion=4.3.5.4, description=null, obTenantId=1002, sqlAuditEnabled=null, arbitrationStatus=DISABLED, role=PRIMARY, serviceNameInfo=null, logTransportMode=null, ocpClusterView=null, basicPrimaryTenant=null, basicStandbyTenantList=null, tags=null, loadType=OLAP, options=null), right=null) 2025-10-27 11:40:27.023 INFO 2458 --- [pool-16-thread-1,,] i.PlanChangePerfDegradationDetectService : diagnose sql stats 2025-10-27 11:40:27.147 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Query plan stat with:QueryPlanRawStatBySql(timeout=30000000, level=0, startTimeUs=1761450027147000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, sqlId=059C3EF76E9644CCC465E3910B3A28F8, obServerId=null, obDbId=201001, dynamicSql=false), context:SqlStatDataContext(clusterId=2, obClusterId=1760510188, clusterName=myoceanbase2, tenantId=6, obTenantId=1002, tenantName=test2, mode=MYSQL, serverId=null, obServerId=null, obVersion=4.3.5.4, startTime=2025-10-26T03:40:27.147Z, endTime=2025-10-27T03:38:26.195Z, parseSqlType=false, parseStatement=false, parseTable=false, obServerIds=[1], serverMap={1=10.40.66.107:2882}, serverIpMap={10.40.66.107=1}, cluster=ObClusterEntity{id=2, name=myoceanbase2, obClusterId=1760510188, status=RUNNING}, tenant=ObTenantEntity{id=6, name=test2, obTenantId=1002, status=NORMAL}, servers=[ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}], server=null) 2025-10-27 11:40:27.187 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Top plan query ret: 259 2025-10-27 11:40:27.187 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan id 2025-10-27 11:40:27.215 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:27.215 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:27.223 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 0 records, use plan hash 2025-10-27 11:40:27.223 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : query plan use plan union hash 2025-10-27 11:40:27.247 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to query plan info 2025-10-27 11:40:27.247 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : queryIncrementBySql with:QueryPlan(timeout=30000000, limit=3000, startTimeUs=1761436800000000, endTimeUs=1761536306195000, obClusterId=1760510188, clusterName=myoceanbase2, obTenantId=1002, obServerId=null, obDbId=201001, sqlId=059C3EF76E9644CCC465E3910B3A28F8, sqlIdList=null, dynamicSql=false) 2025-10-27 11:40:27.252 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.i.PlanRawStatServiceImpl : Begin to map to models for 1 records,use plan union hash 2025-10-27 11:40:27.268 INFO 2458 --- [plan_change_detector1,,] c.o.o.p.s.d.i.a.PlanBindingAdvisor : Got 1 plans with param:QueryTopPlanParam(interval=Interval{start=2025-10-26T11:40:27.147, end=2025-10-27T11:38:26.195}, clusterId=2, tenantId=6, sqlId=059C3EF76E9644CCC465E3910B3A28F8, serverId=null, dbName=oceanbase, obDbId=201001, attachDiagnosis=false, attachPlanExplain=true, attachOutlineBindingStatus=false, dynamicSql=false) 2025-10-27 11:40:27.268 INFO 2458 --- [pool-16-thread-1,,] c.o.o.p.s.d.i.AbstractDiagnoseService : PLAN_CHANGE_DETECTOR, Found out 0 dubious sqls 2025-10-27 11:40:27.445 INFO 2458 --- [ocp-schedules-25,39b5622a8c317edb,be7fb37dfd1cb9d4] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:27.461 WARN 2458 --- [scanner-112229106-1,39b5622a8c317edb,8e735f0fd55709a2] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:27.571 INFO 2458 --- [alarm-detect-1,39b5622a8c317edb,202569941b62f558] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536400 ~ 1761536417 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536415, value=79.683)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536415, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536415=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:27.750 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:27.751 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:27.866 INFO 2458 --- [ocp-schedules-22,cafe2ed27a21ca60,6664ae0b5691928b] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:28.003 INFO 2458 --- [ocp-schedules-19,21c2fdd75808dc8c,861f28569ebb6b55] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:40:28.012 INFO 2458 --- [ocp-schedules-19,21c2fdd75808dc8c,861f28569ebb6b55] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:28.012 INFO 2458 --- [ocp-schedules-19,21c2fdd75808dc8c,861f28569ebb6b55] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:28.547 INFO 2458 --- [ocp-async-10,d1a6696b4f952c53,df9a027db4bf06f4] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:28.698 INFO 2458 --- [ocp-async-4,85bc4996cad79175,b0ce855f441e12f6] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:28.698 WARN 2458 --- [ocp-async-7,b2b66a7286c28e8d,9cc8d23d46b04ef3] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:29.122 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.o.monitor.schedule.MonitorSchedules : measurementIdkey sync key id start. 2025-10-27 11:40:29.122 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=838174751970500608 2025-10-27 11:40:29.124 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.o.monitor.schedule.MonitorSchedules : SeriesIdKey sync key id start. 2025-10-27 11:40:29.124 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=839265989532524544 2025-10-27 11:40:29.126 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.o.monitor.schedule.MonitorSchedules : recordIdKey sync key id start. 2025-10-27 11:40:29.126 INFO 2458 --- [ocp-schedules-29,8af809345e53a6e8,1e33cde646e5f221] c.o.ocp.monitor.service.IdKeyService : Sync key and id, currentMaxSeriesId=835729944023080961 2025-10-27 11:40:29.571 INFO 2458 --- [ocp-schedules-8,893b28615aab0acd,b321946927216956] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info 2025-10-27 11:40:29.587 INFO 2458 --- [ocp-schedules-8,893b28615aab0acd,b321946927216956] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Invalidate clusters [] cache info success. 2025-10-27 11:40:29.587 INFO 2458 --- [ocp-schedules-8,893b28615aab0acd,b321946927216956] c.o.o.c.cache.ConfigServerCacheHolder : [ConfigServerCache]:Refresh all config server cache info success, cached size: 2 2025-10-27 11:40:29.707 INFO 2458 --- [prometheus-minute-schedule-1,a06031130d571ff9,5bc7e288965dc836] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:29.718 INFO 2458 --- [prometheus-minute-schedule-1,a06031130d571ff9,5bc7e288965dc836] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:29.796 INFO 2458 --- [ocp-schedules-10,f852964cf755635f,465c7e58a56b9531] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:29.800 INFO 2458 --- [ocp-schedules-10,f852964cf755635f,465c7e58a56b9531] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:30.698 INFO 2458 --- [ocp-async-4,b21f5c14ab02c522,ece0407f92e1abad] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:30.698 WARN 2458 --- [ocp-async-8,b6e50f133c035ee2,93ecacb9fd64d33d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:30.751 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:30.752 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:31.717 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:31.716172+08:00, lastProcessTime=2025-10-27T11:40:21.713511+08:00 2025-10-27 11:40:31.915 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:31.716172+08:00 2025-10-27 11:40:32.534 INFO 2458 --- [ocp-schedules-14,37f4ceb18b01a00a,8d92c631b16b7d41] c.o.o.m.service.SeriesIdCleanService : Clean series by list started. 2025-10-27 11:40:32.534 INFO 2458 --- [ocp-schedules-14,37f4ceb18b01a00a,8d92c631b16b7d41] c.o.o.m.service.SeriesIdCleanService : Clean series by list finished. 2025-10-27 11:40:32.534 INFO 2458 --- [ocp-schedules-14,37f4ceb18b01a00a,8d92c631b16b7d41] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache started, seriesIdKeySize=8776 2025-10-27 11:40:32.698 WARN 2458 --- [ocp-async-4,c32b3860e9c66fcf,0483045e1c887283] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:32.698 INFO 2458 --- [ocp-async-4,be44ce9c7f0c9d91,993201c517a57774] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:32.707 INFO 2458 --- [prometheus-minute-schedule-1,5fe8848438948ded,ee397e249fe590a3] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:32.714 INFO 2458 --- [prometheus-minute-schedule-1,5fe8848438948ded,ee397e249fe590a3] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:32.807 INFO 2458 --- [ocp-schedules-14,37f4ceb18b01a00a,8d92c631b16b7d41] c.o.o.m.service.SeriesIdCleanService : Clean series info in cache ended, seriesIdKeySize=8776 2025-10-27 11:40:33.752 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:33.753 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:34.044 INFO 2458 --- [ocp-schedules-20,0b7b7e218d9765e3,823f289fc1e61086] c.o.o.s.c.scheduled.TenantSchedules : Check all tenant service name conflict 2025-10-27 11:40:34.044 INFO 2458 --- [ocp-schedules-20,0b7b7e218d9765e3,823f289fc1e61086] .t.CheckAllTenantServiceNameConflictTask : Check all tenant service name conflict. 2025-10-27 11:40:34.698 WARN 2458 --- [ocp-async-4,89ad8fe1511260f1,99ee841b49e4baf2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:34.698 INFO 2458 --- [ocp-async-6,43ec06e1ce0eecb4,2edc06c759aa7cb2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:35.707 INFO 2458 --- [prometheus-minute-schedule-1,49ab040a583af2bf,7ee79ef818313b92] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:35.714 INFO 2458 --- [prometheus-minute-schedule-1,49ab040a583af2bf,7ee79ef818313b92] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:36.698 WARN 2458 --- [ocp-async-6,c30e379ee1fd4108,6b821bab96f4e2a4] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:36.698 INFO 2458 --- [ocp-async-8,88fc2e4a7eac9008,186118a7f2eff90a] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:36.754 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:36.754 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:37.577 INFO 2458 --- [ocp-schedules-8,9f63b7ee510a23dd,e6ed47f29ba6cb28] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:37.616 WARN 2458 --- [scanner-112229304-1,9f63b7ee510a23dd,c5b29fc5ccb8d0e4] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:37.676 INFO 2458 --- [alarm-detect-2,9f63b7ee510a23dd,7dc5f4861ef36d44] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_host_load1_per_cpu_over_threshold detect time range 1761536400 ~ 1761536427 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={load1_per_cpu=[SeriesMetricValues(metric=Metric{name='load1_per_cpu', labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146}}, data=[ValueData(timestamp=1761536415, value=1.552)])]}, metricEvaluateResults={load1_per_cpu > 1.5=[ExpressionEvaluateResult(matchValue=1.5, metric=load1_per_cpu, match=true, value=1.552, values={1761536415=1.552}, valueCount=1, labels={app=HOST, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, obzone=zone1, svr_ip=10.40.65.146})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:37.709 INFO 2458 --- [prometheus-cache-monitor1,2e2469dedabc6fb4,40ecc377eeaf60f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : OcpCacheMonitor, cacheInfo={"secondReadCacheSize":20323,"minuteWriteQueue":0,"minuteReadCacheSize":7854,"secondWriteCacheSize":5080,"seriesIdKeySize":8776,"secondWriteQueue":0,"minuteWriteCacheSize":7894} 2025-10-27 11:40:37.709 INFO 2458 --- [prometheus-cache-monitor1,2e2469dedabc6fb4,40ecc377eeaf60f8] c.o.o.m.s.OcpCacheMonitorServiceImpl : MetricSearchStats:{binlog_instance_convert_fetch_rps=10, ob_sysstat_40012=24, ob_server_resource_memory_assigned_bytes=14, ob_sysstat_40011=24, ob_sysstat_40010=24, ob_unit_num=14, ob_database_disk_required_size=43, go_memstats_mspan_sys_bytes=18, ob_sysstat_30213=24, go_memstats_stack_sys_bytes=18, ob_sysstat_30212=24, ob_sysstat_30217=24, ob_sysstat_30216=24, ob_tenant_request_queue_total_size=29, jvm_gc_memory_allocated_bytes_total=2, ob_system_event_total_waits=33, ob_tenant_disk_v4_log_disk_size=19, ob_sysstat_40019=24, ob_sysstat_40018=24, processor_process_metrics_total=22, jvm_gc_max_data_size_bytes=2, ob_sysstat_40027=24, ob_sysstat_40026=24, ob_sysstat_40025=24, go_memstats_frees_total=18, host_available=13, ob_sysstat_30220=24, ob_sysstat_30222=24, ob_sysstat_30221=24, mysql_output_write_sql_failed_count=17, binlog_instance_network_rx_bytes=11, ob_database_disk_data_size=42, ob_tenant_memstore_freeze_cnt=17, ob_tenant_unit_min_memory=36, ob_sysstat_40028=24, ob_sysstat_60087=24, ob_memstore_total_bytes=29, jvm_threads_states_threads=8, process_resident_memory_bytes=18, ob_cache_size_bytes=40, ob_tenant_unit_max_memory=36, input_collect_metrics_total=26, ob_sysstat_50039=24, ob_sysstat_60083=24, ob_sysstat_50038=24, node_disk_io_time_weighted_seconds_total=45, process_open_fds=18, ob_sysstat_50037=24, process_memory_rss_bytes=32, ob_sysstat_50030=24, ob_server_resource_cpu_assigned_percent=14, sql_audit_input_collect_lost_total=18, go_memstats_heap_inuse_bytes=18, ob_sysstat_40118=24, ob_sysstat_40117=24, node_memory_memfree_bytes=23, ob_sysstat_40116=24, ob_sysstat_40005=24, process_exists=42, ob_sysstat_40004=24, ob_sysstat_40003=24, ob_sysstat_40002=24, binlog_instance_convert_delay=11, go_gc_duration_seconds=23, ob_server_resource_disk_bytes=14, ob_sysstat_40001=24, ob_sysstat_40000=24, binlog_instance_release=11, mysql_output_write_sql_time=25, ob_sysstat_30200=24, odp_current_session=15, ob_sysstat_30201=24, ob_sysstat_30204=24, ob_sysstat_30205=24, ob_sysstat_30208=24, process_thread_count=32, ob 2025-10-27 11:40:37.879 INFO 2458 --- [ocp-schedules-12,d1ec56a89fa04f7b,9c8d76fb06471334] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:38.547 INFO 2458 --- [ocp-async-2,e80c434898b69475,5c52f155af92c854] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:38.698 INFO 2458 --- [ocp-async-10,1eaa76124dda8798,46e1fe3d27620976] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:38.698 WARN 2458 --- [ocp-async-3,02fb3a0a98c5bc0b,ca5025455720562e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:38.707 INFO 2458 --- [prometheus-minute-schedule-1,4672c7ba2cada91a,c07f843f49f519d5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:38.714 INFO 2458 --- [prometheus-minute-schedule-1,4672c7ba2cada91a,c07f843f49f519d5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:39.722 INFO 2458 --- [ocp-schedules-22,1e46265790d84bd1,21d70fb641693856] c.o.o.s.c.scheduled.ClusterSchedules : Collect all cluster compaction info. 2025-10-27 11:40:39.731 INFO 2458 --- [ocp-schedules-22,1e46265790d84bd1,21d70fb641693856] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:39.731 INFO 2458 --- [Collect compaction for all clusters1,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:40:39.732 INFO 2458 --- [Collect compaction for all clusters2,,] s.t.b.c.CollectAllClustersCompactionTask : Ob version >= 4.0.0.0, skip collect compaction 2025-10-27 11:40:39.732 INFO 2458 --- [ocp-schedules-22,1e46265790d84bd1,21d70fb641693856] .o.o.s.t.b.c.AbstractShardingClusterTask : Collect compaction for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:40:39.742 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,8ced622c407c0bff,a36cbbdfe6641347] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=8ced622c407c0bff, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:40:39.742 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-8,8ced622c407c0bff,a36cbbdfe6641347] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=8ced622c407c0bff, duration=1 ms] 2025-10-27 11:40:39.755 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:39.756 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:39.810 INFO 2458 --- [ocp-schedules-26,b85ef26e12f0127f,cd11fef8d505b2fc] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:39.814 INFO 2458 --- [ocp-schedules-26,b85ef26e12f0127f,cd11fef8d505b2fc] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:40.008 INFO 2458 --- [ocp-schedules-30,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:40:40.008 INFO 2458 --- [ocp-schedules-6,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:40:40.046 INFO 2458 --- [Check obproxy1,4bd12419109f8fb4,f63112c8ba349efa] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:40:40.046 INFO 2458 --- [Check obproxy1,4bd12419109f8fb4,f63112c8ba349efa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:40:40.048 INFO 2458 --- [Collect obproxy1,259333a96a2a5fa0,4dc7afa470bd4130] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:40:40.048 INFO 2458 --- [Collect obproxy1,259333a96a2a5fa0,4dc7afa470bd4130] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:40:40.076 INFO 2458 --- [Check obproxy1,4bd12419109f8fb4,f63112c8ba349efa] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:40:40.078 INFO 2458 --- [Collect obproxy1,259333a96a2a5fa0,4dc7afa470bd4130] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:40:40.079 INFO 2458 --- [Check obproxy1,4bd12419109f8fb4,f63112c8ba349efa] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:40:40.079 INFO 2458 --- [Check obproxy1,4bd12419109f8fb4,f63112c8ba349efa] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:40:40.081 INFO 2458 --- [Collect obproxy1,259333a96a2a5fa0,4dc7afa470bd4130] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:40:40.084 INFO 2458 --- [Collect obproxy1,259333a96a2a5fa0,4dc7afa470bd4130] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:40:40.699 INFO 2458 --- [ocp-async-7,8ab2388462a3fbe6,b2c283b384f28fcc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:40.699 WARN 2458 --- [ocp-async-5,b6ad99547c367c1e,686fa89402f4844f] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:40.826 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:40.826 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=3 2025-10-27 11:40:40.826 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=2 2025-10-27 11:40:40.826 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=1 2025-10-27 11:40:40.826 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=4 2025-10-27 11:40:40.826 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Get OCP agent alarms: hostId=5 2025-10-27 11:40:40.836 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:40.838 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:40.839 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:40.839 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:40.840 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:40.841 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:40.842 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:40.842 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:40.843 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:40.843 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:40.897 INFO 2458 --- [ForkJoinPool.commonPool-worker-4,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62889/api/v1/log/alarms, params:null 2025-10-27 11:40:40.898 INFO 2458 --- [ForkJoinPool.commonPool-worker-1,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62889/api/v1/log/alarms, params:null 2025-10-27 11:40:40.898 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62889/api/v1/log/alarms, params:null 2025-10-27 11:40:40.899 INFO 2458 --- [ForkJoinPool.commonPool-worker-3,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62889/api/v1/log/alarms, params:null 2025-10-27 11:40:40.900 INFO 2458 --- [ForkJoinPool.commonPool-worker-7,,] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62889/api/v1/log/alarms, params:null 2025-10-27 11:40:40.902 INFO 2458 --- [ocp-schedules-12,0547a150d4970ced,e4412b417aec5553] c.o.o.s.common.scheduled.HostSchedules : Get host agent alarms finished. 2025-10-27 11:40:41.317 INFO 2458 --- [ocp-schedules-17,e2b8e28e3fbb101b,d50e181240bcef90] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog instances, size=0 2025-10-27 11:40:41.567 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.o.s.c.scheduled.TenantSchedules : Check suspend tenant compaction status 2025-10-27 11:40:41.567 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] s.t.b.t.CheckSuspendTenantCompactionTask : Check tenant compaction status. 2025-10-27 11:40:41.671 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:41.672 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:40:41.687 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:41.687 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ 100 * (1- SUM(UNFINISHED_TABLET_COUNT) / SUM(TOTAL_TABLET_COUNT)) FROM GV$OB_COMPACTION_PROGRESS WHERE TENANT_ID = ? AND COMPACTION_SCN = ? 2025-10-27 11:40:41.707 INFO 2458 --- [prometheus-minute-schedule-1,0608c5d82882c59d,8634f6ecccb39ca1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:41.714 INFO 2458 --- [prometheus-minute-schedule-1,0608c5d82882c59d,8634f6ecccb39ca1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:41.720 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:41.718868+08:00, lastProcessTime=2025-10-27T11:40:31.716172+08:00 2025-10-27 11:40:41.766 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:41.767 INFO 2458 --- [ocp-schedules-7,393e398e64631602,f56421b3d37a7ca2] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ tenant_id, global_broadcast_scn AS broadcast_scn, is_error AS error, status, frozen_scn, last_scn, is_suspended AS suspend, info, start_time, last_finish_time FROM CDB_OB_MAJOR_COMPACTION 2025-10-27 11:40:41.937 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:41.718868+08:00 2025-10-27 11:40:42.699 WARN 2458 --- [ocp-async-7,84f77e15408c593d,957259fb2d47b9b0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:42.699 INFO 2458 --- [ocp-async-2,e3ce0d46f7a1e79f,211768431ab72765] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:42.756 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:42.757 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:44.366 INFO 2458 --- [ocp-schedules-26,04937da24c9aaa53,e12bb3e45ab8d391] c.o.o.s.c.scheduled.ClusterSchedules : Check certificate expired. 2025-10-27 11:40:44.699 WARN 2458 --- [ocp-async-3,37049545dd0fe622,e350c91f2f7bf9c2] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:44.699 INFO 2458 --- [ocp-async-1,723d1cc860af2522,e051070adcdce79d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:44.707 INFO 2458 --- [prometheus-minute-schedule-1,0ebe7ba49a97c6d4,d10f1a86e39929b8] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:44.714 INFO 2458 --- [prometheus-minute-schedule-1,0ebe7ba49a97c6d4,d10f1a86e39929b8] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:45.102 INFO 2458 --- [prometheus-second-schedule-1,8c9fc833a4c59e60,eeca6f0b838741f3] c.o.ocp.monitor.OcpMonitorManager : create pull exporter, exporterCount=15, collectTaskCount=15 2025-10-27 11:40:45.757 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:45.758 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:46.699 WARN 2458 --- [ocp-async-5,9a630bf0e66e717c,8d526851389e9cbe] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:46.699 INFO 2458 --- [ocp-async-5,87d37477d653e032,7291ac46fedc4372] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:47.707 INFO 2458 --- [prometheus-minute-schedule-1,97ab795e20d1fb55,9f443f8f361049b4] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:47.715 INFO 2458 --- [prometheus-minute-schedule-1,97ab795e20d1fb55,9f443f8f361049b4] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:47.773 INFO 2458 --- [ocp-schedules-21,4143887f80a10ebc,c0285c1b82745ecc] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:47.782 WARN 2458 --- [scanner-112229516-1,4143887f80a10ebc,6ab2e979a40c34d7] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:47.886 INFO 2458 --- [ocp-schedules-4,ee3b151db8ac4d6f,a1e069de833f0ca2] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:47.894 INFO 2458 --- [alarm-detect-13,4143887f80a10ebc,3221fb1d8a29ef0d] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536415 ~ 1761536437 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536430, value=82.498)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536430, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536430=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:48.014 INFO 2458 --- [ocp-schedules-1,d81524c92e34adbe,058f54e0e453bbcd] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:40:48.023 INFO 2458 --- [ocp-schedules-1,d81524c92e34adbe,058f54e0e453bbcd] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:48.023 INFO 2458 --- [ocp-schedules-1,d81524c92e34adbe,058f54e0e453bbcd] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:40:48.547 INFO 2458 --- [ocp-async-1,908db3fc6eaa2d12,f888297a9c0c0cbd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:48.699 WARN 2458 --- [ocp-async-7,187b45a0207205e8,cd49636d4dd5b2a1] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:48.699 INFO 2458 --- [ocp-async-6,3da96365fcb9b782,3ae3eddf86741d9c] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:48.758 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:48.760 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:49.822 INFO 2458 --- [ocp-schedules-21,7929a33962122e19,6392af6c9efa849a] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:49.826 INFO 2458 --- [ocp-schedules-21,7929a33962122e19,6392af6c9efa849a] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:40:50.455 INFO 2458 --- [ocp-schedules-25,9c7cecef1d09af87,b4f901b3bb57c83e] c.o.o.s.common.scheduled.PerfSchedules : Collect slow sql for alarm, time=2025-10-27T11:40:50.455+08:00, clusters=[1, 2] 2025-10-27 11:40:50.502 INFO 2458 --- [ocp-schedules-25,9c7cecef1d09af87,b4f901b3bb57c83e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase for slowSqlStat, sql count is 0. 2025-10-27 11:40:50.505 INFO 2458 --- [ocp-schedules-25,9c7cecef1d09af87,b4f901b3bb57c83e] c.o.o.p.s.i.alarm.SlowSqlDataSource : query sqlCount of obCluster myoceanbase2 for slowSqlStat, sql count is 0. 2025-10-27 11:40:50.699 WARN 2458 --- [ocp-async-2,eda376f251715b05,6e425158c0d3ce7e] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:50.699 INFO 2458 --- [ocp-async-4,8504ea447f498cbf,5b67550fccd965ab] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:50.707 INFO 2458 --- [prometheus-minute-schedule-1,ba230ed740c8ab76,bb1b699db80489bb] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:50.714 INFO 2458 --- [prometheus-minute-schedule-1,ba230ed740c8ab76,bb1b699db80489bb] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:51.410 INFO 2458 --- [ocp-schedules-24,a814a872531c5787,12ede1f08a0e36cc] c.o.o.s.common.scheduled.HostSchedules : Check host agent status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:51.410 INFO 2458 --- [ocp-schedules-13,c654ca73d2ae4ac2,72721818c0fb98bd] c.o.o.s.common.scheduled.HostSchedules : Check host status schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:51.410 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:40:51.410 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=1, ip=10.40.65.146, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=2, ip=10.40.65.168, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=3, ip=10.40.67.22, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.h.manager.HostCheckManagerImpl : Check host agent status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:40:51.411 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=4, ip=10.40.65.151, originalStatus=AVAILABLE 2025-10-27 11:40:51.412 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status, hostId=5, ip=10.40.66.107, originalStatus=ONLINE 2025-10-27 11:40:51.424 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:51.425 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:51.425 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:51.426 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:51.426 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:51.426 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:51.427 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:51.428 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:51.428 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:51.430 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:51.431 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:51.433 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:51.434 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:51.434 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:51.436 INFO 2458 --- [ocp-schedules-10,77c84f219698ca89,ea24a4196887c3ff] c.o.o.s.common.scheduled.HostSchedules : Check host clockdiff schedule, hostIps=[10.40.65.146, 10.40.65.168, 10.40.67.22, 10.40.65.151, 10.40.66.107] 2025-10-27 11:40:51.436 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:51.436 INFO 2458 --- [host-check-clockdiff1,77c84f219698ca89,3ab7cb0c15724617] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.146], timeoutMillis=20000 2025-10-27 11:40:51.436 INFO 2458 --- [host-check-clockdiff2,77c84f219698ca89,4395b59d8c02a3b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.168], timeoutMillis=20000 2025-10-27 11:40:51.436 INFO 2458 --- [host-check-clockdiff3,77c84f219698ca89,70ae43ecb211d82f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.67.22], timeoutMillis=20000 2025-10-27 11:40:51.439 INFO 2458 --- [host-check-clockdiff4,77c84f219698ca89,dbfe9592fb5b8602] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.65.151], timeoutMillis=20000 2025-10-27 11:40:51.439 INFO 2458 --- [host-check-clockdiff5,77c84f219698ca89,e9f298412fd44f2d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, executable=clockdiff, args=[10.40.66.107], timeoutMillis=20000 2025-10-27 11:40:51.445 INFO 2458 --- [ocp-schedules-26,039e413ff7c50bf2,f5435c0c797c9444] c.o.o.s.c.scheduled.TenantSchedules : Check all standby tenant 2025-10-27 11:40:51.457 INFO 2458 --- [host-check-clockdiff2,77c84f219698ca89,4395b59d8c02a3b8] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536451 -1 -1 , error= 2025-10-27 11:40:51.449 INFO 2458 --- [host-check-clockdiff4,77c84f219698ca89,dbfe9592fb5b8602] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536451 0 0 , error= 2025-10-27 11:40:51.474 INFO 2458 --- [host-check-clockdiff1,77c84f219698ca89,3ab7cb0c15724617] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536451 0 0 , error= 2025-10-27 11:40:51.474 INFO 2458 --- [host-check-clockdiff5,77c84f219698ca89,e9f298412fd44f2d] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536451 0 0 , error= 2025-10-27 11:40:51.521 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/agent/status, params:null 2025-10-27 11:40:51.521 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/agent/status, params:null 2025-10-27 11:40:51.521 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/time, params:null 2025-10-27 11:40:51.526 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.67.22:62888/api/v1/time, params:null 2025-10-27 11:40:51.526 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.66.107 2025-10-27 11:40:51.526 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.66.107 2025-10-27 11:40:51.526 INFO 2458 --- [host-check-agent-status5,a814a872531c5787,3c38f04a9e50f458] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.66.107 2025-10-27 11:40:51.526 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=1, ip=10.40.65.146, timestamp=2025-10-27T11:40:51.526621493+08:00 2025-10-27 11:40:51.527 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.146:62888/api/v1/agent/status, params:null 2025-10-27 11:40:51.527 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.67.22 2025-10-27 11:40:51.527 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.67.22 2025-10-27 11:40:51.527 INFO 2458 --- [host-check-agent-status3,a814a872531c5787,b1bfac038c7c4578] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.67.22 2025-10-27 11:40:51.528 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=3, ip=10.40.67.22, timestamp=2025-10-27T11:40:51.527406478+08:00 2025-10-27 11:40:51.528 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=3, lastAvailableTime=2025-10-27T11:40:51.528+08:00 2025-10-27 11:40:51.528 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=1, lastAvailableTime=2025-10-27T11:40:51.526+08:00 2025-10-27 11:40:51.530 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.146 2025-10-27 11:40:51.530 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.146 2025-10-27 11:40:51.530 INFO 2458 --- [host-check-agent-status1,a814a872531c5787,8e50da03fb20896f] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.146 2025-10-27 11:40:51.534 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.66.107:62888/api/v1/time, params:null 2025-10-27 11:40:51.535 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=5, ip=10.40.66.107, timestamp=2025-10-27T11:40:51.535192136+08:00 2025-10-27 11:40:51.535 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/time, params:null 2025-10-27 11:40:51.536 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.s.compute.HostAgentDaoManager : Updated agent 2 last available time 2025-10-27T11:40:51.528+08:00 and status AVAILABLE 2025-10-27 11:40:51.537 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=2, ip=10.40.65.168, timestamp=2025-10-27T11:40:51.535927687+08:00 2025-10-27 11:40:51.537 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.s.compute.HostAgentDaoManager : Updated agent 3 last available time 2025-10-27T11:40:51.526+08:00 and status AVAILABLE 2025-10-27 11:40:51.537 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=5, lastAvailableTime=2025-10-27T11:40:51.535+08:00 2025-10-27 11:40:51.537 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=2, lastAvailableTime=2025-10-27T11:40:51.537+08:00 2025-10-27 11:40:51.538 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.168:62888/api/v1/agent/status, params:null 2025-10-27 11:40:51.539 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/agent/status, params:null 2025-10-27 11:40:51.541 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.168 2025-10-27 11:40:51.541 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.168 2025-10-27 11:40:51.541 INFO 2458 --- [host-check-agent-status2,a814a872531c5787,65226d02e308845a] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.168 2025-10-27 11:40:51.542 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=3 2025-10-27 11:40:51.543 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=1 2025-10-27 11:40:51.544 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 5 last available time 2025-10-27T11:40:51.535+08:00 and status AVAILABLE 2025-10-27 11:40:51.545 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_agentd status=running, hostIp=10.40.65.151 2025-10-27 11:40:51.545 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mgragent status=running, hostIp=10.40.65.151 2025-10-27 11:40:51.545 INFO 2458 --- [host-check-agent-status4,a814a872531c5787,041002934df0b8dd] c.o.o.c.h.manager.HostCheckManagerImpl : Host agent ocp_mongent status=running, hostIp=10.40.65.151 2025-10-27 11:40:51.545 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.s.compute.HostAgentDaoManager : Updated agent 1 last available time 2025-10-27T11:40:51.537+08:00 and status AVAILABLE 2025-10-27 11:40:51.545 INFO 2458 --- [ocp-schedules-17,7a9302977858ba62,456176670e0c7942] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster status schedule. 2025-10-27 11:40:51.548 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.e.internal.template.HttpTemplate : GET request to agent, url:http://10.40.65.151:62888/api/v1/time, params:null 2025-10-27 11:40:51.548 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.manager.HostCheckManagerImpl : Check host status result. id=4, ip=10.40.65.151, timestamp=2025-10-27T11:40:51.548248544+08:00 2025-10-27 11:40:51.548 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=5 2025-10-27 11:40:51.549 INFO 2458 --- [ocp-schedules-12,3a4d8cfb61d39ffb,f5c0bf361da7d78a] c.o.o.s.c.scheduled.ClusterSchedules : Check standby cluster info schedule. 2025-10-27 11:40:51.549 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.s.compute.HostAgentDaoManager : Updating OCP agent last available time with status as available, hostId=4, lastAvailableTime=2025-10-27T11:40:51.548+08:00 2025-10-27 11:40:51.549 INFO 2458 --- [ocp-schedules-17,7a9302977858ba62,456176670e0c7942] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:51.553 INFO 2458 --- [ocp-schedules-12,3a4d8cfb61d39ffb,f5c0bf361da7d78a] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:51.553 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=2 2025-10-27 11:40:51.554 INFO 2458 --- [ocp-schedules-22,9298b700b4a29bc9,9cc5cca633d09360] c.o.o.s.c.scheduled.ClusterSchedules : Sync cluster info schedule. 2025-10-27 11:40:51.555 INFO 2458 --- [ocp-schedules-12,3a4d8cfb61d39ffb,f5c0bf361da7d78a] .o.o.s.t.b.c.AbstractShardingClusterTask : Check all standby clusters sync delay time executed completed, allCount=2, successCount=2 2025-10-27 11:40:51.557 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.s.compute.HostAgentDaoManager : Updated agent 4 last available time 2025-10-27T11:40:51.548+08:00 and status AVAILABLE 2025-10-27 11:40:51.559 INFO 2458 --- [ocp-schedules-22,9298b700b4a29bc9,9cc5cca633d09360] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:40:51.562 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.manager.HostCheckManagerImpl : Refreshing OCP agent process running info: hostId=4 2025-10-27 11:40:51.563 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:51.564 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=3 2025-10-27 11:40:51.566 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:51.566 INFO 2458 --- [ocp-schedules-25,5dd0f76de12a785c,d898a70f1c97a223] c.o.o.s.c.scheduled.ClusterSchedules : Check all arbitration services 2025-10-27 11:40:51.567 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 3 2025-10-27 11:40:51.568 INFO 2458 --- [ocp-schedules-15,ba83aba181ef16c1,7614a33b756f91c9] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=BackupAlarmCheckSchedule, method=run 2025-10-27 11:40:51.569 INFO 2458 --- [ocp-schedules-18,8b36a03f6fb1ff74,2d5aab622ed57542] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] begin refresh method to audit event meta map... 2025-10-27 11:40:51.570 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:51.573 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:51.575 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=2 2025-10-27 11:40:51.579 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 2 2025-10-27 11:40:51.581 INFO 2458 --- [ocp-schedules-15,ba83aba181ef16c1,7614a33b756f91c9] c.o.o.b.i.alarm.BackupAlarmService : Backup alarm check schedule. 2025-10-27 11:40:51.582 INFO 2458 --- [ocp-schedules-18,8b36a03f6fb1ff74,2d5aab622ed57542] c.o.ocp.service.audit.AuditEventService : [refreshMethodToAuditEventMetaIdMap] refresh method to audit event meta map success 2025-10-27 11:40:51.589 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=4 2025-10-27 11:40:51.590 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.590 INFO 2458 --- [Check status for all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:51.591 INFO 2458 --- [Check status for all clusters2,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=2, obVersion=4.3.5.4 2025-10-27 11:40:51.592 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 4 2025-10-27 11:40:51.611 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.611 INFO 2458 --- [Check status for all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:51.616 INFO 2458 --- [Check status for all clusters1,,] c.o.o.s.t.b.c.CheckClusterStatusTask : check cluster status okay, clusterId=1, obVersion=4.3.5.4 2025-10-27 11:40:51.616 INFO 2458 --- [ocp-schedules-17,7a9302977858ba62,456176670e0c7942] .o.o.s.t.b.c.AbstractShardingClusterTask : Check status for all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:40:51.623 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:51.625 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:51.627 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:51.630 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.630 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:51.635 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:51.643 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=3 2025-10-27 11:40:51.644 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=13 2025-10-27 11:40:51.650 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_mgragent, skipPorts=false), params:null 2025-10-27 11:40:51.656 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=2 2025-10-27 11:40:51.658 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=1 2025-10-27 11:40:51.665 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=10 2025-10-27 11:40:51.668 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.669 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:51.678 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=3 2025-10-27 11:40:51.683 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:51.683 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.684 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:40:51.685 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=13 2025-10-27 11:40:51.686 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=2 2025-10-27 11:40:51.693 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=10 2025-10-27 11:40:51.700 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=1 2025-10-27 11:40:51.707 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:51.713 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:51.714 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:51.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:40:51.714332+08:00, lastProcessTime=2025-10-27T11:40:41.718868+08:00 2025-10-27 11:40:51.725 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:51.728 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_monagent, skipPorts=false), params:null 2025-10-27 11:40:51.729 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=14 2025-10-27 11:40:51.732 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=6 2025-10-27 11:40:51.741 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:51.741 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=11 2025-10-27 11:40:51.741 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.742 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [cluster] 2025-10-27 11:40:51.743 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:51.743 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.744 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=4 2025-10-27 11:40:51.744 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:51.745 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.748 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:40:51.749 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:51.760 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:51.761 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:51.764 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=5 2025-10-27 11:40:51.774 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=14 2025-10-27 11:40:51.779 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=6 2025-10-27 11:40:51.779 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=4, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:51.779 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:30+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:51.784 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=4 2025-10-27 11:40:51.785 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=11 2025-10-27 11:40:51.804 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:51.805 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=5 2025-10-27 11:40:51.805 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=4, ip=10.40.66.107, port=2882, sqlPort=2881, installPath=/home/admin/oceanbase, runPath=/home/admin/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=5, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-17T14:41:29.605727+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:51.812 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.67.22:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:51.819 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.151:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:51.821 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.168:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:51.823 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=15 2025-10-27 11:40:51.834 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=12 2025-10-27 11:40:51.836 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=9 2025-10-27 11:40:51.837 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/process/info, request body:GetProcessInfoRequest(processName=ocp_agentd, skipPorts=false), params:null 2025-10-27 11:40:51.839 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:51.839 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.840 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ `zone`, `status`, `region`, `idc` FROM `DBA_OB_ZONES` 2025-10-27 11:40:51.841 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=7 2025-10-27 11:40:51.841 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.845 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ zone, svr_ip, svr_port, sql_port as inner_port, with_rootserver, UPPER(`status`) as `status`, build_version, UNIX_TIMESTAMP(NVL(`stop_time`, 0))*1000000 as stop_time, UNIX_TIMESTAMP(NVL(`start_service_time`, 0))*1000000 as start_service_time, UNIX_TIMESTAMP(NVL(`block_migrate_in_time`, 0))*1000000 as block_migrate_in_time, id FROM DBA_OB_SERVERS ORDER BY zone ASC, id ASC 2025-10-27 11:40:51.846 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:51.847 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.848 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:51.849 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase2 2025-10-27 11:40:51.860 INFO 2458 --- [host-check-host-status5,c654ca73d2ae4ac2,80335e00f637104c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=15 2025-10-27 11:40:51.863 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 2025-10-27 11:40:51.869 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.h.s.HostProcessServiceImpl : Refreshing process running info: processId=8 2025-10-27 11:40:51.873 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=5 2025-10-27 11:40:51.876 INFO 2458 --- [Sync all clusters2,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 5 2025-10-27 11:40:51.877 INFO 2458 --- [host-check-host-status4,c654ca73d2ae4ac2,80902f9dae75ad8a] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=12 2025-10-27 11:40:51.878 INFO 2458 --- [host-check-host-status2,c654ca73d2ae4ac2,d1421d09c8fbe32c] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=7 2025-10-27 11:40:51.885 INFO 2458 --- [host-check-host-status3,c654ca73d2ae4ac2,37a92c9a18992044] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=9 2025-10-27 11:40:51.896 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=2, name=zone3, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:51.897 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:51.906 INFO 2458 --- [host-check-host-status1,c654ca73d2ae4ac2,cc41bce350a38c18] c.o.o.c.a.p.AbstractProcessOperator : Refreshing process running info: id=8 2025-10-27 11:40:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=2, ip=10.40.65.168, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=2, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.090726+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:51.942 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:51.949 INFO 2458 --- [Sync all clusters2,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.66.107:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:40:51.958 INFO 2458 --- [ocp-schedules-1,2128ad5d7f10c8c6,a29281ac8d686eae] c.o.o.s.c.scheduled.BinlogSchedules : Sync binlog clusters, size=0 2025-10-27 11:40:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=3, name=zone2, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:51.978 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:51.983 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:40:51.714332+08:00 2025-10-27 11:40:51.992 INFO 2458 --- [Sync all clusters2,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 5, sqlPort: 2881 got ObProcessInfoResult(pid=28211, runPath=/home/admin/oceanbase, runUser=admin, execute=/home/admin/oceanbase/bin/observer, installPath=/home/admin/oceanbase) 2025-10-27 11:40:51.992 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.993 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:40:51.995 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:51.996 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:40:52.011 WARN 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:40:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=3, ip=10.40.67.22, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=false, hostId=3, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.747940+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:52.017 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605203] detect zone changed, before=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, current=ObZoneEntity{id=null, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null} 2025-10-27 11:40:52.035 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase2, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:52.036 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:52.037 INFO 2458 --- [Sync all clusters2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:40:52.052 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] zone updated, zone=ObZoneEntity{id=1, name=zone1, idcId=1, status=RUNNING, innerStatus=ACTIVE, compactionStatus=null, startTime=null, stopTime=null}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:52.052 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605303] detect server changed, before=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:03+08:00, stopTime=1970-01-01T08:00+08:00}, current=ObServerEntity{id=null, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=null, runPath=null, runUser=null, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00} 2025-10-27 11:40:52.061 INFO 2458 --- [Sync all clusters2,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=2, clusterName=myoceanbase2, obClusterId=1760510188 2025-10-27 11:40:52.061 INFO 2458 --- [Sync all clusters2,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=2, name=myoceanbase2, obClusterId=1760510188, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=CREATED, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase2","ObClusterId":1760510188,"ObRegion":"myoceanbase2","ObRegionId":1760510188,"ReadonlyRsList":[],"RsList":[{"address":"10.40.66.107:2882","role":"LEADER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761535884668000}, createTime=2025-10-17T14:35:58+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=el7, loadType=OLAP, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"DEFAULT","InstallPath":"/home/admin/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=2) 2025-10-27 11:40:52.089 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] server updated, server=ObServerEntity{id=1, ip=10.40.65.146, port=2882, sqlPort=2881, installPath=/home/admin/myoceanbase/oceanbase, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, version=4.3.5.4_104000042025090916-5cf5b925a25bf888aebaa288e251b85b1924e98a(Sep 9 2025 17:07:42), withRootserver=true, hostId=1, status=RUNNING, innerStatus=ACTIVE, startTime=2025-10-15T14:40:02.710584+08:00, stopTime=1970-01-01T08:00+08:00}, oldStatus=RUNNING, newStatus=RUNNING 2025-10-27 11:40:52.127 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:52.127 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:52.128 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ max(value) value from oceanbase.__all_virtual_sys_parameter_stat where name = 'min_observer_version' 2025-10-27 11:40:52.131 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : Update cluster attributes info of cluster myoceanbase 2025-10-27 11:40:52.142 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 2025-10-27 11:40:52.156 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.agent.HostAgentServiceImpl : Finding OCP agent: hostId=1 2025-10-27 11:40:52.159 INFO 2458 --- [Sync all clusters1,,] c.o.o.c.a.p.HostAgentProcessServiceImpl : Getting all OCP agent processes on host 1 2025-10-27 11:40:52.204 INFO 2458 --- [Sync all clusters1,,] c.o.o.e.internal.template.HttpTemplate : POST request to agent, url:http://10.40.65.146:62888/api/v1/ob/getProcessInfo, request body:ObproxyProcessInfoRequest(sqlPort=2881), params:null 2025-10-27 11:40:52.268 INFO 2458 --- [Sync all clusters1,,] .o.o.i.c.ClusterHostOperationServiceImpl : get ob process info host: 1, sqlPort: 2881 got ObProcessInfoResult(pid=10271, runPath=/home/admin/myoceanbase/oceanbase, runUser=admin, execute=/home/admin/myoceanbase/oceanbase/bin/observer, installPath=/home/admin/myoceanbase/oceanbase) 2025-10-27 11:40:52.268 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:52.269 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ COUNT(1) FROM information_schema.tables WHERE table_schema = 'oceanbase' AND table_name = '__all_license' 2025-10-27 11:40:52.271 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:52.272 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ PRODUCT_TYPE, END_USER, LICENSE_ID, LICENSE_CODE, LICENSE_TYPE,ISSUANCE_DATE, ACTIVATION_TIME, EXPIRED_TIME,OPTIONS,NODE_NUM FROM oceanbase.__all_license LIMIT 1 2025-10-27 11:40:52.307 WARN 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] Not a dual primary cluster, found 1. 2025-10-27 11:40:52.345 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.o.o.f.ConnectPropertiesBuilder : get credential from obsdk context, clusterName=myoceanbase, tenantName=sys, dbUser=ocp_monitor 2025-10-27 11:40:52.346 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:40:52.347 INFO 2458 --- [Sync all clusters1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SELECT /*+ ocp_query */ arbitration_service_key, arbitration_service, previous_arbitration_service, type FROM DBA_OB_ARBITRATION_SERVICE 2025-10-27 11:40:52.386 INFO 2458 --- [Sync all clusters1,,] c.o.o.o.i.c.ClusterSyncServiceImpl : [OCP605] sync cluster done, clusterId=1, clusterName=myoceanbase, obClusterId=1760510187 2025-10-27 11:40:52.386 INFO 2458 --- [Sync all clusters1,,] c.o.o.s.t.b.cluster.SyncAllClustersTask : [OCP605] cluster sync over, doSync=true, cluster=ObClusterOverview(id=1, name=myoceanbase, obClusterId=1760510187, vpcId=1, vpcName=system-default, type=PRIMARY, status=RUNNING, origin=TAKEN_OVER, syncStatus=VALID, rootserverJson={"ObCluster":"myoceanbase","ObClusterId":1760510187,"ObRegion":"myoceanbase","ObRegionId":1760510187,"ReadonlyRsList":[],"RsList":[{"address":"10.40.65.146:2882","role":"LEADER","sql_port":2881},{"address":"10.40.65.168:2882","role":"FOLLOWER","sql_port":2881},{"address":"10.40.67.22:2882","role":"FOLLOWER","sql_port":2881}],"Type":"PRIMARY","timestamp":1761536402504000}, createTime=2025-10-15T16:37:43+08:00, creator=admin, obVersion=4.3.5.4, packageOperatingSystem=null, loadType=null, attributesJson={"DataDiskPath":"/opt/oceanbase/data","DiskPathStyle":"OBD","InstallPath":"/home/admin/myoceanbase/oceanbase","LogDiskPath":"/opt/oceanbase/log","OperatingSystemUser":"admin","RunPath":"/home/admin/myoceanbase/oceanbase","SqlPort":2881,"SvrPort":2882}, tenantCount=4) 2025-10-27 11:40:52.386 INFO 2458 --- [ocp-schedules-22,9298b700b4a29bc9,9cc5cca633d09360] .o.o.s.t.b.c.AbstractShardingClusterTask : Sync all clusters executed completed, allCount=2, successCount=2 2025-10-27 11:40:52.699 WARN 2458 --- [ocp-async-1,d821c975813a7f02,2109fe1d9bb7f359] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:52.699 INFO 2458 --- [ocp-async-9,d219896d8ca5bcb7,c1d36a0f345c0bc1] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:53.165 INFO 2458 --- [print-time-series-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Async compute queue max size = 0, current size = 0. 2025-10-27 11:40:53.223 INFO 2458 --- [host-check-clockdiff3,77c84f219698ca89,70ae43ecb211d82f] c.o.ocp.core.util.LocalCommandExecutor : Execute local command, exitCode=0, out=1761536453 0 0 , error= 2025-10-27 11:40:53.707 INFO 2458 --- [prometheus-minute-schedule-1,aa5ba87b33e0a01d,56f848aee4bb3721] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:53.714 INFO 2458 --- [prometheus-minute-schedule-1,aa5ba87b33e0a01d,56f848aee4bb3721] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:54.699 WARN 2458 --- [ocp-async-7,ea518e46c3c67a8a,10c01231980cfbb0] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:54.699 INFO 2458 --- [ocp-async-2,f418072d29273ecd,6de8d51d80106538] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:54.761 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:54.762 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:56.000 INFO 2458 --- [ocp-schedules-8,7992c85dcdb2459f,10feb079c51f9aa9] c.o.o.alarm.service.OcpAlarmServiceImpl : fix alarm target, alarms size 58 2025-10-27 11:40:56.592 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,adf6f7c2537bcac9,cfc1fb1d60ae668d] c.o.o.s.c.trace.RequestTracingAspect : API: [GET /services?User_ID=alibaba&UID=test&Action=GetObProxyConfig&ObproxyClusterName=obproxyTest&VersionOnly=true, client=10.40.67.22, traceId=adf6f7c2537bcac9, method=CommonResult com.oceanbase.ocp.server.common.controllerv22.OcpControllerConfig.getObProxyConfig(String,String,String,String,boolean), args=alibaba,test,,obproxyTest,true,] 2025-10-27 11:40:56.592 INFO 2458 --- [http-nio-0.0.0.0-8081-exec-2,adf6f7c2537bcac9,cfc1fb1d60ae668d] c.o.o.s.c.trace.RequestTracingAspect : API OK: [GET /services client=10.40.67.22, traceId=adf6f7c2537bcac9, duration=1 ms] 2025-10-27 11:40:56.700 WARN 2458 --- [ocp-async-3,8b5dc8a914fbc967,b3f1d5fe65fa164d] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:56.700 INFO 2458 --- [ocp-async-4,27bd7043dd98a597,25199917d6abb7a2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:56.707 INFO 2458 --- [prometheus-minute-schedule-1,9e951f6a3972ed8c,a6e18280002fd309] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:56.714 INFO 2458 --- [prometheus-minute-schedule-1,9e951f6a3972ed8c,a6e18280002fd309] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:57.762 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:40:57.763 INFO 2458 --- [pool-8-thread-1,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:40:57.891 INFO 2458 --- [ocp-schedules-8,2f246a8a02498387,62158f4aa3af37ba] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:40:57.900 INFO 2458 --- [ocp-schedules-25,8e4b4ac497da2272,204db6669a2bf6c5] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:40:57.938 WARN 2458 --- [scanner-112229723-1,8e4b4ac497da2272,bf12393b14659a12] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:40:58.001 INFO 2458 --- [alarm-detect-3,8e4b4ac497da2272,f67196e7c05fb1c3] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_compaction detect time range 1761536387 ~ 1761536447 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_compaction_seconds=[SeriesMetricValues(metric=Metric{name='no_compaction_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536400, value=204734.56)])]}, metricEvaluateResults={no_compaction_seconds > 108000=[ExpressionEvaluateResult(matchValue=108000.0, metric=no_compaction_seconds, match=true, value=204734.56, values={1761536400=204734.56}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:40:58.025 INFO 2458 --- [alarm-detect-20,8e4b4ac497da2272,14647f1762d40db6] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_log_disk_usage_high detect time range 1761536388 ~ 1761536448 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={ob_tenant_log_disk_usage=[SeriesMetricValues(metric=Metric{name='ob_tenant_log_disk_usage', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536400, value=93.281)])]}, metricEvaluateResults={ob_tenant_log_disk_usage > 85=[ExpressionEvaluateResult(matchValue=85.0, metric=ob_tenant_log_disk_usage, match=true, value=93.281, values={1761536400=93.281}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:40:58.031 INFO 2458 --- [alarm-detect-3,8e4b4ac497da2272,1f6e5400dc0d505b] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_no_frozen detect time range 1761536388 ~ 1761536448 matched, alarm-level: 2, evaluate result MultiExpressionEvaluateResult(metricSnapshots={no_frozen_seconds=[SeriesMetricValues(metric=Metric{name='no_frozen_seconds', labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536400, value=204735.912)])]}, metricEvaluateResults={no_frozen_seconds > 90000=[ExpressionEvaluateResult(matchValue=90000.0, metric=no_frozen_seconds, match=true, value=204735.912, values={1761536400=204735.912}, valueCount=1, labels={app=OB, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=2) 2025-10-27 11:40:58.074 INFO 2458 --- [alarm-detect-13,8e4b4ac497da2272,a9dd426e548cd342] c.o.o.alarm.core.detect.MetricDetector : alarmType ob_tenant_request_queue_over_threshold detect time range 1761536430 ~ 1761536448 matched, alarm-level: 3, evaluate result MultiExpressionEvaluateResult(metricSnapshots={request_queue_time=[SeriesMetricValues(metric=Metric{name='request_queue_time', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536445, value=92.19)])], ob_tenant_request_queue_size=[SeriesMetricValues(metric=Metric{name='ob_tenant_request_queue_size', labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor}}, data=[ValueData(timestamp=1761536445, value=17409.0)])]}, metricEvaluateResults={ob_tenant_request_queue_size > 100=[ExpressionEvaluateResult(matchValue=100.0, metric=ob_tenant_request_queue_size, match=true, value=17409.0, values={1761536445=17409.0}, valueCount=1, labels={app=OB, host=10.40.65.146, host_name=tjtx40-65-146.zhuanos.com, ob_cluster_id=1760510187, ob_cluster_name=myoceanbase, ob_tenant_id=1004, obzone=zone1, svr_ip=10.40.65.146, tenant_name=ocp_monitor})]}, matched=true, isContainValidValue=null, message=null, level=3) 2025-10-27 11:40:58.547 INFO 2458 --- [ocp-async-10,7588b59eda1f1fc1,5e60987e21f6a248] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:40:58.700 WARN 2458 --- [ocp-async-8,08a02b76100767eb,6e5760c710fc2245] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:40:58.700 INFO 2458 --- [ocp-async-1,5433efc7ab18f787,deca5882bca94ef0] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:40:59.655 INFO 2458 --- [ocp-schedules-8,09ad3392c7e1ed3f,e184fa9a50291635] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration auto replace status 2025-10-27 11:40:59.707 INFO 2458 --- [prometheus-minute-schedule-1,13e3497632e867d6,cf48e5a049c5b082] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536460, interval=60 2025-10-27 11:40:59.714 INFO 2458 --- [prometheus-minute-schedule-1,13e3497632e867d6,cf48e5a049c5b082] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:40:59.834 INFO 2458 --- [ocp-schedules-9,788ec50127d8cadc,de4dbdd799760e4b] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:40:59.838 INFO 2458 --- [ocp-schedules-9,788ec50127d8cadc,de4dbdd799760e4b] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:41:00.700 INFO 2458 --- [ocp-async-7,87c59da0b861eb53,b552a521e77a582e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:00.700 WARN 2458 --- [ocp-async-2,9870c13275a73d8b,edd26f3e109825cb] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:00.763 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:41:00.764 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:41:01.715 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:41:01.714526+08:00, lastProcessTime=2025-10-27T11:40:51.714332+08:00 2025-10-27 11:41:01.952 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:41:01.714526+08:00 2025-10-27 11:41:02.700 INFO 2458 --- [ocp-async-7,177769cd15f3a66d,a3827e205f6be10f] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:02.700 WARN 2458 --- [ocp-async-3,f2fc7fca8c23a720,ece02b014666b826] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:02.707 INFO 2458 --- [prometheus-minute-schedule-1,b678fb6a6079db28,f02079e0c6fd028c] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536520, interval=60 2025-10-27 11:41:02.716 INFO 2458 --- [prometheus-minute-schedule-1,b678fb6a6079db28,f02079e0c6fd028c] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:41:03.764 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:41:03.765 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:41:04.700 WARN 2458 --- [ocp-async-5,0adf63286d4712d2,06ea14f3522bad96] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:04.700 INFO 2458 --- [ocp-async-9,81416f5ab7ba25f2,7610a1d94618aecc] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:05.359 INFO 2458 --- [ocp-schedules-25,5f9879d080367db3,7c8cb71c4367545b] c.o.o.s.c.scheduled.ClusterSchedules : Check cluster zone cgroup value. 2025-10-27 11:41:05.367 INFO 2458 --- [ocp-schedules-25,5f9879d080367db3,7c8cb71c4367545b] .o.o.s.t.b.c.AbstractShardingClusterTask : Consume clusters: [1, 2] 2025-10-27 11:41:05.406 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:41:05.407 INFO 2458 --- [Check cluster cgroup value.2,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:41:05.420 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: set ob_query_timeout = ?, args: [10000000] 2025-10-27 11:41:05.421 INFO 2458 --- [Check cluster cgroup value.1,,] c.o.ocp.obsdk.connector.ConnectTemplate : [obsdk] sql: SHOW PARAMETERS WHERE scope = 'cluster' AND name LIKE ? , args: [enable_cgroup] 2025-10-27 11:41:05.435 INFO 2458 --- [ocp-schedules-25,5f9879d080367db3,7c8cb71c4367545b] .o.o.s.t.b.c.AbstractShardingClusterTask : Check cluster cgroup value. executed completed, allCount=2, successCount=2 2025-10-27 11:41:05.707 INFO 2458 --- [prometheus-minute-schedule-1,ad8cdd674977fb5c,6a9a26fea3a922a2] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536520, interval=60 2025-10-27 11:41:05.714 INFO 2458 --- [prometheus-minute-schedule-1,ad8cdd674977fb5c,6a9a26fea3a922a2] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:41:06.700 WARN 2458 --- [ocp-async-2,db86f113e3607e2a,e4bd5ed14d474431] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:06.700 INFO 2458 --- [ocp-async-6,3c9a66dd708944d2,36a50353aa9995f3] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:06.766 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:41:06.767 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:41:07.898 INFO 2458 --- [ocp-schedules-4,594f95523a48a0e6,894dd2ad5095f393] c.o.ocp.monitor.OcpMonitorManager : Refresh cached second exporters 2025-10-27 11:41:08.025 INFO 2458 --- [ocp-schedules-1,39600ee2f7ae8768,92d297036740d7b4] c.o.o.a.service.TaskRepositoryService : markNoHeartbeatToFailed success, affectRow=0 2025-10-27 11:41:08.034 INFO 2458 --- [ocp-schedules-1,39600ee2f7ae8768,92d297036740d7b4] c.o.o.a.service.TaskRepositoryService : heartbeat done, affectRow=1, taskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:41:08.034 INFO 2458 --- [ocp-schedules-1,39600ee2f7ae8768,92d297036740d7b4] c.o.o.alarm.utils.task.AlarmTaskManager : updateHeartbeat, tryCount=1, heartbeatCount=1, currentTaskUuidList=[a5ee0103-d3db-461c-81b9-a863a3328b5e] 2025-10-27 11:41:08.079 INFO 2458 --- [ocp-schedules-28,87f23db283f6f645,5ea94009682bc0e9] c.o.ocp.alarm.core.task.AlarmDetectTask : alarm detect start 2025-10-27 11:41:08.096 WARN 2458 --- [scanner-112229952-1,87f23db283f6f645,7e7db629b5e2b837] c.o.o.m.helper.MetricSearchContainer : label is null, return all seriesIds, metricName = node_net_bandwidth_bps, seriesId size = 12 2025-10-27 11:41:08.162 INFO 2458 --- [alarm-detect-14,87f23db283f6f645,1eba19945d1a6ef2] c.o.o.a.c.detect.AlarmRuleDetectState : skipPending, alarmType=ob_host_load1_per_cpu_over_threshold, target=alarm_template_id=0:host=10.40.65.146-3, activeAt=null 2025-10-27 11:41:08.537 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.s.common.scheduled.PerfSchedules : Collect high risk sql audit stat, time=2025-10-27T11:41:08.537+08:00, tenantSize=6 2025-10-27 11:41:08.547 INFO 2458 --- [ocp-async-9,4127c6302b69ff17,53feec0dcaea593e] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSnapshot, method=observerAlarmEvents 2025-10-27 11:41:08.584 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6currentTimeUs:1761536468537000 2025-10-27 11:41:08.584 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6diagnosisStartTimeUs:1761536168501000 2025-10-27 11:41:08.622 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_2_6totalLength:0 2025-10-27 11:41:08.635 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4currentTimeUs:1761536468627000 2025-10-27 11:41:08.635 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4diagnosisStartTimeUs:1761536168409000 2025-10-27 11:41:08.683 INFO 2458 --- [ocp-schedules-20,ff45a2ed7629f685,6c694422292373bb] c.o.o.p.s.i.HighRiskSqlInspectService : taskName:high_risk_sql_audit_diag_1_4totalLength:0 2025-10-27 11:41:08.700 WARN 2458 --- [ocp-async-7,dff45d79cf597c74,07ef1000c7d6c280] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:08.700 INFO 2458 --- [ocp-async-7,2812ed132576cb63,6d589d8a4aa5d744] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:08.707 INFO 2458 --- [prometheus-minute-schedule-1,8583e0f969dbcd29,538e95e081e9eba5] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536520, interval=60 2025-10-27 11:41:08.714 INFO 2458 --- [prometheus-minute-schedule-1,8583e0f969dbcd29,538e95e081e9eba5] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:41:09.767 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:41:09.768 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:41:09.846 INFO 2458 --- [ocp-schedules-19,b13ca19f5aa3996c,773baa7b7f216c69] c.o.o.s.c.scheduled.ClusterSchedules : Check arbitration service group 2025-10-27 11:41:09.850 INFO 2458 --- [ocp-schedules-19,b13ca19f5aa3996c,773baa7b7f216c69] o.i.a.t.CheckArbitrationServiceGroupTask : No ob server lose connection with its arbitration 2025-10-27 11:41:10.008 INFO 2458 --- [ocp-schedules-29,,] c.o.o.s.c.scheduled.ObproxySchedules : Collect obproxy metrics, size=1 2025-10-27 11:41:10.008 INFO 2458 --- [ocp-schedules-26,,] c.o.o.s.c.scheduled.ObproxySchedules : Check obproxy server stats, size=1 2025-10-27 11:41:10.041 INFO 2458 --- [Check obproxy1,e1c21386a1923eb8,ad5cf471e2291d01] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:41:10.041 INFO 2458 --- [Check obproxy1,e1c21386a1923eb8,ad5cf471e2291d01] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:41:10.046 INFO 2458 --- [Collect obproxy1,9741300e5a6e7ca2,2b90f8008978b58c] c.o.o.s.o.SystemOdpOperatorFactory : connect obproxy 10.40.67.22 2883 2025-10-27 11:41:10.046 INFO 2458 --- [Collect obproxy1,9741300e5a6e7ca2,2b90f8008978b58c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig, args: [] 2025-10-27 11:41:10.077 INFO 2458 --- [Check obproxy1,e1c21386a1923eb8,ad5cf471e2291d01] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [proxy_id] 2025-10-27 11:41:10.080 INFO 2458 --- [Check obproxy1,e1c21386a1923eb8,ad5cf471e2291d01] c.o.o.o.internal.ObproxyServiceImpl : check obproxy 1 success, update last active time to now 2025-10-27 11:41:10.080 INFO 2458 --- [Check obproxy1,e1c21386a1923eb8,ad5cf471e2291d01] c.o.o.o.internal.ObproxyServiceImpl : Obproxy 1 proxy_id: 1, need to sync to meta: false 2025-10-27 11:41:10.082 INFO 2458 --- [Collect obproxy1,9741300e5a6e7ca2,2b90f8008978b58c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%proxy_mem_limited%] 2025-10-27 11:41:10.085 INFO 2458 --- [Collect obproxy1,9741300e5a6e7ca2,2b90f8008978b58c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%max_connections%] 2025-10-27 11:41:10.087 INFO 2458 --- [Collect obproxy1,9741300e5a6e7ca2,2b90f8008978b58c] com.oceanbase.ocp.odpsdk.OdpOperator : [odpsdk] sql: SHOW proxyconfig LIKE ?, args: [%client_max_connections%] 2025-10-27 11:41:10.180 INFO 2458 --- [async-queue-offer-interval5-1,,] c.o.o.m.a.MonitorAsyncComputeProcessor : Offer async queue, interval = 5, queue size = 17, ruleSize = 17. 2025-10-27 11:41:10.701 WARN 2458 --- [ocp-async-1,8d7e652d51c5e164,510637b39717e902] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:10.701 INFO 2458 --- [ocp-async-8,e053348dfaefed40,f76c539228ddebe2] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:11.707 INFO 2458 --- [prometheus-minute-schedule-1,30328e3087df308a,a76c32d828093108] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536520, interval=60 2025-10-27 11:41:11.714 INFO 2458 --- [prometheus-minute-schedule-1,30328e3087df308a,a76c32d828093108] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14 2025-10-27 11:41:11.714 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process alarm start, currentTimestamp=2025-10-27T11:41:11.713874+08:00, lastProcessTime=2025-10-27T11:41:01.714526+08:00 2025-10-27 11:41:11.921 INFO 2458 --- [alarm-task-1,4cdd20a261ddee19,d9accaf8bc4623b5] c.o.o.alarm.core.process.AlarmProcessor : process done, currentTimestamp=2025-10-27T11:41:11.713874+08:00 2025-10-27 11:41:12.593 INFO 2458 --- [metric-archive-async-5,,] c.o.o.m.service.OcpMetricArchiveService : async type metric archive, timestamp=1761536400, nodeCount=821, affectRows=821 2025-10-27 11:41:12.701 WARN 2458 --- [ocp-async-8,053d2d0ecf415b4f,96e14479f11fe967] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:12.701 INFO 2458 --- [ocp-async-9,45f3213939195799,e791c0dfb687c6cd] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:12.768 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.s.RateLimitFacadeServiceImpl : 10.40.65.146:8081 refreshes policies... 2025-10-27 11:41:12.769 INFO 2458 --- [pool-8-thread-2,,] c.o.o.s.i.r.util.RateLimitCounter : After refresh, size of counterEntries :0 2025-10-27 11:41:13.044 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536400, nodeCount=1000, affectRows=1000 2025-10-27 11:41:13.048 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536400, nodeCount=994, validCount=992 2025-10-27 11:41:13.120 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536400, nodeCount=1000, affectRows=1000 2025-10-27 11:41:13.124 INFO 2458 --- [metric-archive-second-7,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536400, nodeCount=999, validCount=995 2025-10-27 11:41:13.137 ERROR 2458 --- [metric-archive-second-7,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134602000) Duplicate entry '835370602992181248-1761536340' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134602000) Duplicate entry '835370602992181248-1761536340' for key 'PRIMARY' 2025-10-27 11:41:13.176 ERROR 2458 --- [metric-archive-second-5,,] c.o.o.m.storage.RollupMetricDataDao : Archive node list to table metric_data_second error, e = PreparedStatementCallback; SQL [INSERT INTO `metric_data_second` (`series_id`, `timestamp`, `data`, `interval`) VALUES (?, ?, ?, ?)]; (conn=134587947) Duplicate entry '835014129065861120-1761536340' for key 'PRIMARY'; nested exception is java.sql.BatchUpdateException: (conn=134587947) Duplicate entry '835014129065861120-1761536340' for key 'PRIMARY' 2025-10-27 11:41:13.200 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536400, nodeCount=80, affectRows=80 2025-10-27 11:41:13.200 INFO 2458 --- [metric-archive-second-6,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536400, nodeCount=79, validCount=79 2025-10-27 11:41:13.247 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Second type metric archive, timestamp=1761536400, nodeCount=1000, affectRows=1000 2025-10-27 11:41:13.250 INFO 2458 --- [metric-archive-second-1,,] c.o.o.m.service.OcpMetricArchiveService : Rollup second to minute, timestamp=1761536400, nodeCount=996, validCount=992 2025-10-27 11:41:14.701 WARN 2458 --- [ocp-async-9,db270423ea6665a5,5e9cdde908231947] c.o.o.a.s.AlarmInMemoryServiceImpl : aggregator or allDistributors is empty. 2025-10-27 11:41:14.701 INFO 2458 --- [ocp-async-4,af47d28fdd0f64f2,2fdd7180d8cfe24d] c.o.o.s.common.DistributedLockAspect : Get distributed lock, lockKey=AlarmEventProcessForSelfcure, method=observerAlarmEvents 2025-10-27 11:41:14.707 INFO 2458 --- [prometheus-minute-schedule-1,301a8883280117ac,53ff11992af2f2c1] c.o.ocp.monitor.OcpMonitorManager : Next exec sec=1761536520, interval=60 2025-10-27 11:41:14.714 INFO 2458 --- [prometheus-minute-schedule-1,301a8883280117ac,53ff11992af2f2c1] c.o.ocp.monitor.OcpMonitorManager : Collect interval metrics, interval=60, exporterCount=14, collectTaskCount=14